Effective Control Transport Inc. (fka EFFC)

[Believer Fever]

I like that..a GAME CHANGER....I am looking forward to hearing Rosebud's comments on the patent:

The device, as described above further comprises an alert connected to the microprocessor for alerting the subject to the microsleep event.

Accordingly, in another aspect there is provided a method of alerting a vehicle operator to a microsleep event, the method comprising: Accordingly, in another aspect there is provided a method of alerting a vehicle operator to a microsleep event, the method comprising:

- determining a plurality of eye openness factors by measuring a plurality of distances between an upper eyelid and a lower eyelid of at least one eye over a time period; - generating graphical representations of the eye openness factors; - Determining a plurality of eye openness factors by measuring a plurality of distances between an upper eyelid and a lower eyelid of at least one eye over a time period; - generating graphical representations of the eye openness factors;

- correlating changes in the eye openness factors over the time period with a reference eye closure pattern indicative of the microsleep event; and - Correlating changes in the eye openness factors over the time period with a reference eye closure pattern indicative of the microsleep event; and

- triggering an alarm so as to alert the operator to the microsleep event. - Triggering an alarm so as to alert the operator to the microsleep event.

Accordingly, in yet another aspect, there is provided a method of correlating EEG and EOG microsleep patterns with eye closure patterns, the method comprising: Accordingly, in yet another aspect, there is provided a method of correlating EEG and EOG microsleep patterns with eye closure patterns, the method comprising:

- measuring EEG and EOG microsleep patterns in a subject; - Measuring EEG and EOG microsleep patterns in a subject;

- determining a plurality of eye openness factors by measuring a plurality of distances between an upper eyelid and a lower eyelid of at least one eye over a time period; - Determining a plurality of eye openness factors by measuring a plurality of distances between an upper eyelid and a lower eyelid of at least one eye over a time period;

- generating graphical representations of the eye openness factors; and - Generating graphical representations of the eye openness factors; and

- correlating changes in the eye openness factors with the EEG and EOG microsleep patterns. - Correlating changes in the eye openness factors with the EEG and EOG microsleep patterns.

BRIEF DESCRIPTION OF THE DRAWINGS BRIEF DESCRIPTION OF THE DRAWINGS



Embodiments of the invention will be described by way of example only with reference to the accompany drawings, in which: Embodiments of the invention will be described by way of example only with reference to the accompany drawings, in which:

Figure 1 is a block diagram of a microsleep detection device according to an illustrative embodiment of the present invention, the device being used on a subject; Figure 1 is a block diagram of a microsleep detection device according to an illustrative embodiment of the present invention, the device being used on a subject;

Figure 2 is a flow diagram of a microsleep event detection process that may be used by the device of Figure 1 ; Figure 2 is a flow diagram of a microsleep event detection process that may be used by the device of Figure 1;

Figure 3A and 3B are schematic views of the fully open eye (Figure 3A) and the fully closed eye (Figure 3B) with their associated eye openness factor; Figure 3A and 3B are schematic views of the fully open eye (Figure 3A) and the fully closed eye (Figure 3B) with their associated eye openness factor;

Figure 4 is a series of schematic views of an example of the variation of the eye openness factor as a function of time for a blink cycle; Figure 4 is a series of schematic views of an example of the variation of the eye openness factor as a function of time for a blink cycle;

Figure 5 is an illustrative example of the fully closed eye (period L4 and L5), at the moment of a microsleep; and Figure 5 is an illustrative example of the fully closed eye (period L4 and L5), at the moment of a microsleep; and

Figure 6 is a flow diagram of a microsleep characteristic eye openness factor levels detection sub-process that may be used with the microsleep event detection process of Figure 2 for an image sampling frequency of 20 Hz. Figure 6 is a flow diagram of a microsleep characteristic eye openness factor levels detection sub-process that may be used with the microsleep event detection process of Figure 2 for an image sampling frequency of 20 Hz.

DETAILED DESCRIPTION Generally stated, the non-limitative illustrative embodiment of the present invention provides a method and device for the detection of microsleep events in a human subject based on the analysis of eye closure patterns of at least one eye, typically both eyes, which occur during microsleep events. DETAILED DESCRIPTION Generally stated, the non-limitative illustrative embodiment of the present invention provides a method and device for the detection of microsleep events in a human subject based on the analysis of eye closure patterns of at least one eye, typically both eyes, which occur during microsleep events.

During microsleep events, measured by EEG and EOG, we observed a progressive variation of the distance between an upper eyelid and a lower eyelid of a human subject over time. We now describe hereinbelow a method and device, which provides complementary evidence of impending sleep in drowsy operators, such as vehicle drivers, airplane pilots, air traffic controllers and the like, using eye closure patterns, which when detected, alerts sleepy operators to unsafe situations before eyelid closure occurs, thereby correlating EEG and EOG microsleep patterns with eye closure patterns. During microsleep events, measured by EEG and EOG, we observed a progressive variation of the distance between an upper eyelid and a lower eyelid of a human subject over time. We now describe hereinbelow a method and device, which provides complementary evidence of impending sleep in drowsy operators, such as vehicle drivers, airplane pilots, air traffic controllers and the like, using eye closure patterns, which when detected, alerts sleepy operators to unsafe situations before eyelid closure occurs, thereby correlating EEG and EOG microsleep patterns with eye closure patterns.

Also described is a method for the analysis of eye closure patterns allowing the differentiation between a normal eyelid closure and one due to drowsiness. Also described is a method for the analysis of eye closure patterns allowing the differentiation between a normal eyelid closure and one due to drowsiness.

Referring to Figure 1 , there is shown an example of a microsleep event detection device 100, which generally comprises facial image sampler such as a digital camera 102 with an associated infrared source 104, a microprocessor 106 with an associated memory 108 and either or both an alarm/display 110 and input/output interface 112. Referring to Figure 1, there is shown an example of a microsleep event detection device 100, which generally comprises facial image sampler such as a digital camera 102 with an associated infrared source 104, a microprocessor 106 with an associated memory 108 and either or both an alarm / display 110 and input / output interface 112.

Thus, in one example, there is described a method using the device 100 for detecting a microsleep event in a human subject. The method comprises: determining a plurality of eye openness factors by measuring a plurality of distances between an upper eyelid and a lower eyelid of at least one eye, typically both eyes, over a time period; generating graphical representations of Thus, in one example, there is described a method using the device 100 for detecting a microsleep event in a human subject. The method comprises: determining a plurality of eye openness factors by measuring a plurality of distances between an upper eyelid and a lower eyelid of at least one eye, typically both eyes, over a time period; generating graphical representations of

the eye openness factors; and correlating changes in the eye openness factors over the time period with a reference eye closure pattern indicative of the microsleep event. the eye openness factors; and correlating changes in the eye openness factors over the time period with a reference eye closure pattern indicative of the microsleep event.

In operation, the digital camera 102 is aimed at the face of a subject 10 and illuminates his or her eyes 12 using the infrared source 104 in order to determine the eye openness factor, ie a value representative of the distance between the upper 14a and lower 14b eyelids. The images taken by the digital camera 102 are then processed by the processor 106 which executes a microsleep event detection process stored on its associated memory 108. Upon detection of a microsleep event, the microsleep event detection device 100 may inform the user of a microsleep event by triggering an integrated alarm and/or display 110 or provide the information to a further process or device via the input/output interface 112. It is to be understood that other components may be added to the microsleep event detection device 100 such as, for example, a user interface and a wireless communication device. In operation, the digital camera 102 is aimed at the face of a subject 10 and illuminates his or her eyes 12 using the infrared source 104 in order to determine the eye openness factor, ie a value representative of the distance between the upper 14a and lower 14b eyelids. The images taken by the digital camera 102 are then processed by the processor 106 which executes a microsleep event detection process stored on its associated memory 108. Upon detection of a microsleep event, the microsleep event detection device 100 may inform the user of a microsleep event by triggering an integrated alarm and / or display 110 or provide the information to a further process or device via the input / output interface 112. It is to be understood that other components may be added to the microsleep event detection device 100 such as, for example, a user interface and a wireless communication device.

Referring now to Figure 2, there is shown a flow diagram of a microsleep event detection process 200 that may be electronically executed by the processor 106 of the microsleep event detection device 100 of Figure 1. The steps of the process 200 are indicated by blocks 202 to 212. The process 200 starts at block 202 by sampling an image of the face of a subject 10 using the digital camera 102. The digital camera 102 may sample images at a frequency between about 10 and 60Hz (ie sampling frequency). At block 204, the process 200 identifies, in the sampled digital image, the eye 12 and eyelids 14a, 14b of the subject 10. This may be accomplished using a facial feature recognition algorithm executed by the processor 106. Then, the eye openness factor is computed. Referring now to Figure 2, there is shown a flow diagram of a microsleep event detection process 200 that may be electronically executed by the processor 106 of the microsleep event detection device 100 of Figure 1. The steps of the process 200 are indicated by blocks 202 to 212. The process 200 starts at block 202 by sampling an image of the face of a subject 10 using the digital camera 102. The digital camera 102 may sample images at a frequency between about 10 and 60Hz (ie sampling frequency). At block 204, the process 200 identifies, in the sampled digital image, the eye 12 and eyelids 14a, 14b of the subject 10. This may be accomplished using a facial feature recognition algorithm executed by the processor 106. Then, the eye openness factor is computed .

Referring now to Figures 3A and 3B, the eye openness factor may be expressed as d having a value between 1 , representing a fully open eye 12 (see Figure 3A), and 0, representing a fully closed eye 12 (see Figure 3B). The value of d may be computed, for example, by dividing the measured distance between the upper Referring now to Figures 3A and 3B, the eye openness factor may be expressed as d having a value between 1, representing a fully open eye 12 (see Figure 3A), and 0, representing a fully closed eye 12 (see Figure 3B). The value of d may be computed, for example, by dividing the measured distance between the upper

14a and lower 14b eyelids positions in a Cartesian representation (X 1 Y) by a reference measure of the fully open eye D. Thus: 14a and lower 14b eyelids positions in a Cartesian representation (X 1 Y) by a reference measure of the fully open eye D. Thus:

d =(upper lid position (x u ,y u ) - lower lid position (xi,y?))/D A d = (upper lid position (x u, y u) - lower lid position (xi, y?)) / D A

where D = upper lid position (xu,yu) - lower lid position (xL.yO and (x u ,y u ) = instant upper lid position, (x?,y?) = instant lower lid position, (xu,yu) = upper lid position at maximum eye opening and (xL.yO = lower lid position at maximum eye opening. where D = upper lid position (xu, yu) - lower lid position (xL.yO and (x u, y u) = instant upper lid position, (x?, y?) = instant lower lid position, (xu, yu) = upper lid position at maximum eye opening and (xL.yO = lower lid position at maximum eye opening.

Referring again to Figure 2 at block 206, the process verifies if the microsleep characteristic eye openness factor levels are present. Referring again to Figure 2 at block 206, the process verifies if the microsleep characteristic eye openness factor levels are present.

Referring to Figure 4, there is shown an illustrative example of the variation of the eye openness factor as a function of time for a blink cycle. The blink cycle starts at time ti with a fully open eye 12 (eye openness factor d = 1.00), at t 2 the eye openness factor remains at d = 1.00, then decreases, going through d = 0.80 and d = 0.50 at times t, and t,+i , until it reaches d = 0.00 at time t j (fully closed eye 12), and then increases, going through d = 0.50 and d = 0.80 at times t k and t k +i, until it reaches d = 1.00 again at t n It is to be understood that the blink cycle illustrated in Figure 4 for illustrative purpose only and that an actual cycle would comprise a number of sample times depending the sample frequency. Referring to Figure 4, there is shown an illustrative example of the variation of the eye openness factor as a function of time for a blink cycle. The blink cycle starts at time ti with a fully open eye 12 (eye openness factor d = 1.00) , at t 2 the eye openness factor remains at d = 1.00, then decreases, going through d = 0.80 and d = 0.50 at times t, and t, + i, until it reaches d = 0.00 at time t j (fully closed eye 12), and then increases, going through d = 0.50 and d = 0.80 at times t k and t k + i, until it reaches d = 1.00 again at t n It is to be understood that the blink cycle illustrated in Figure 4 for illustrative purpose only and that an actual cycle would comprise a number of sample times depending the sample frequency.

Referring now to Figure 5, these computed eye openness factors, as a function of time, can be represented in a graph which may be generally characterized by five successive levels, L1 to L5. The first and last levels, L1 and L5, are associated with the open eye (ie d = 1.00), the second level, L2, with the closure of the eyelids, 14a, 14b (ie 0.00 > d > 0.00, d decreasing), the third level L3, with the partial or closed eye (ie 0.00 < d < 0.5 for example) and fourth level, L4, with the opening of the eyelids 14a, 14b (ie 0.00 < d < 1.00, d increasing). Referring now to Figure 5, these computed eye openness factors, as a function of time, can be represented in a graph which may be generally characterized by five successive levels, L1 to L5. The first and last levels, L1 and L5, are associated with the open eye (ie d = 1.00), the second level, L2, with the closure of the eyelids, 14a, 14b (ie 0.00> d> 0.00, d decreasing), the third level L3, with the partial or closed eye (ie 0.00 <d <0.5 for example) and fourth level, L4, with the opening of the eyelids 14a, 14b (ie 0.00 <d <1.00, d increasing).

Referring again to Figure 2, the process 200 detects all five levels, ie L1 to L5, it then proceeds to block 208. If not, it proceeds back to block 202 for the next Referring again to Figure 2, the process 200 detects all five levels, ie L1 to L5, it then proceeds to block 208. If not, it proceeds back to block 202 for the next

image sample. At block 208, the process computes the eye closure 21 or 26 and eye opening 25 or 27 representative curves. The eye closure representative curve 21 or 26 is computed using negative slope second order polynomial regression (parabolic curve), ie image sample. At block 208, the process computes the eye closure 21 or 26 and eye opening 25 or 27 representative curves. The eye closure representative curve 21 or 26 is computed using negative slope second order polynomial regression (parabolic curve), ie

Y = d o + d 1 .X + d 2 .X 2 . Y = d o + d 1. X + d 2. X 2.

where Y is the predicted outcome value for the polynomial model with regression coefficients di to 2 for each degree and Y intercept d1 ; where Y is the predicted outcome value for the polynomial model with regression coefficients di to 2 for each degree and Y intercept d1;

which is applied to the eye openness factors composing the first and second eye openness factor levels, ie L1 and L2. As for the eye opening representative curve 25 or 27, it is computed using positive slope second order polynomial regression applied to the eye openness factors composing the fourth and fifth eye openness factor levels, ie L4 and L5. Then, at block 210, the process 220 verifies if the microsleep eye opening and closing representative curves are present. This is accomplished by computing the Pearson coefficient, r: which is applied to the eye openness factors composing the first and second eye openness factor levels, ie L1 and L2. As for the eye opening representative curve 25 or 27, it is computed using positive slope second order polynomial regression applied to the eye openness factors composing the fourth and fifth eye openness factor levels, ie L4 and L5. Then, at block 210, the process 220 verifies if the microsleep eye opening and closing representative curves are present. This is accomplished by computing the Pearson coefficient, r:



where X and Y are positions in a Cartesian representation; where X and Y are positions in a Cartesian representation;

of the eye closure representative curve 21 or 26 with regard to the eye openness factors composing eye openness factor levels L1 and L2, and of the eye opening representative curve 25 or 27 with regard to the eye openness factors composing the eye openness factor levels L4 and L5. If both Pearson coefficients are greater or equal to a given threshold such as, for example, 0.9, then the process 200 proceeds to block 212. If not, it proceeds back to block 202 for next image sample. Finally, at block 212, the microsleep event detection device 100 may inform the user 10 of the detection of a microsleep event state via the integrated alarm and/or display 110 or provide the information to a further process or device via the input/output interface 112 (see Figure 1) using, for instance, a wired or of the eye closure representative curve 21 or 26 with regard to the eye openness factors composing eye openness factor levels L1 and L2, and of the eye opening representative curve 25 or 27 with regard to the eye openness factors composing the eye openness factor levels L4 and L5. If both Pearson coefficients are greater or equal to a given threshold such as, for example, 0.9, then the process 200 proceeds to block 212. If not, it proceeds back to block 202 for next image sample. Finally, at block 212 , the microsleep event detection device 100 may inform the user 10 of the detection of a microsleep event state via the integrated alarm and / or display 110 or provide the information to a further process or device via the input / output interface 112 (see Figure 1 ) using, for instance, a wired or

wireless telecommunication link such as, for example, Bluetooth, WiFi and the like. wireless telecommunication link such as, for example, Bluetooth, WiFi and the like.

It is to be understood that the Pearson coefficient threshold is not meant to be restricted to 0.9 and may be adjusted to suit a desired confidence level. It may also vary depending on the resolution of the digital camera 102 (see Figure 1). It is to be understood that the Pearson coefficient threshold is not meant to be restricted to 0.9 and may be adjusted to suit a desired confidence level. It may also vary depending on the resolution of the digital camera 102 (see Figure 1).

The eye closure pattern is based on these particular observations closing, duration of the eyelid complete or partial closure and re-opening. More precisely, the eye closure pattern indicates a progressive decreasing followed by a baseline period where the eyelids are fully closed and then a reopening. If all the above occur, then a microsleep is detected. The eye closure pattern is based on these particular observations closing, duration of the eyelid complete or partial closure and re-opening. More precisely, the eye closure pattern indicates a progressive decreasing followed by a baseline period where the eyelids are fully closed and then a reopening. If all the above occur, then a microsleep is detected.

Example of microsleep characteristic eye openness factor levels at an image sampling frequency of 20GHz Example of microsleep characteristic eye openness factor levels at an image sampling frequency of 20GHz

Referring now to Figure 6, there is shown a flow diagram a microsleep characteristic eye openness factor levels detection sub-process 300 that may be executed at block 206 of process 200 in order to detect the presence of the five microsleep characteristic eye openness factor levels, ie L1 to L5 (see Figure 5) for an image sampling frequency of 20 Hz. The steps of the sub-process 300 are indicated by blocks 301 to 305. Referring now to Figure 6, there is shown a flow diagram a microsleep characteristic eye openness factor levels detection sub-process 300 that may be executed at block 206 of process 200 in order to detect the presence of the five microsleep characteristic eye openness factor levels, ie L1 to L5 (see Figure 5) for an image sampling frequency of 20 Hz. The steps of the sub-process 300 are indicated by blocks 301 to 305.

At block 301 , the sub-process 300 verifies if the first level L1 is detected. To that end, the sub-process 300 checks if a series of a minimum of six (6) successive eye openness factors having a value of d = 1.00 is present. If so, the sub-process 300 proceeds to block 302, if not, it goes back to block 202 of process 200 (see Figure 2). At block 301, the sub-process 300 verifies if the first level L1 is detected. To that end, the sub-process 300 checks if a series of a minimum of six (6) successive eye openness factors having a value of d = 1.00 is present. If so, the sub-process 300 proceeds to block 302, if not, it goes back to block 202 of process 200 (see Figure 2).

At block 302, the sub-process 300 verifies if the second level L2 is detected. To that end, the sub-process 300 checks if a series of a minimum of four (4) successive decreasing eye openness factors having values between of d = 0.99 At block 302, the sub-process 300 verifies if the second level L2 is detected. To that end, the sub-process 300 checks if a series of a minimum of four (4) successive decreasing eye openness factors having values between of d = 0.99

and d = 0.01 is present. If so, the sub-process 300 proceeds to block 303, if not, it goes back to block 202 of process 200 (see Figure 2). and d = 0.01 is present. If so, the sub-process 300 proceeds to block 303, if not, it goes back to block 202 of process 200 (see Figure 2).

Then, at block 303, the sub-process 300 verifies if the third level L3 is detected. To that end, the sub-process 300 checks if a series of a minimum of five (5) and a maximum of 120 successive eye openness factors having a value of d = 0.00 is present. If so, the sub-process 300 proceeds to block 304, if not, it goes back to block 202 of process 200 (see Figure 2). Then, at block 303, the sub-process 300 verifies if the third level L3 is detected. To that end, the sub-process 300 checks if a series of a minimum of five (5) and a maximum of 120 successive eye openness factors having a value of d = 0.00 is present. If so, the sub-process 300 proceeds to block 304, if not, it goes back to block 202 of process 200 (see Figure 2).

At block 304, the sub-process 300 verifies if the fourth level L4 is detected. To that end, the sub-process 300 checks if a series of a minimum of four (4) successive increasing eye openness factors having values between of d = 0.01 and d = 0.99 is present. If so, the sub-process 300 proceeds to block 305, if not, it goes back to block 202 of process 200 (see Figure 2). At block 304, the sub-process 300 verifies if the fourth level L4 is detected. To that end, the sub-process 300 checks if a series of a minimum of four (4) successive increasing eye openness factors having values between of d = 0.01 and d = 0.99 is present. If so, the sub-process 300 proceeds to block 305, if not, it goes back to block 202 of process 200 (see Figure 2).

Finally, at block 305, the sub-process 300 verifies if the fifth level L5 is detected. To that end, the sub-process 300 checks if a series of a minimum of six (6) successive eye openness factors having a value of d = 1.00 is present. If so, the sub-process 300 proceeds to block 208 of process 200 (see Figure 2), the presence of all five (5) microsleep characteristic eye openness factor levels. If not, the sub-process 300 goes back to block 202 of process 200 (see Figure 2). It is to be understood that the number of eye openness factors used to detect the presence of each microsleep characteristic eye openness factor level may vary, for example according to the image sampling frequency and are meant as illustrative examples only. Finally, at block 305, the sub-process 300 verifies if the fifth level L5 is detected. To that end, the sub-process 300 checks if a series of a minimum of six (6) successive eye openness factors having a value of d = 1.00 is present. If so, the sub-process 300 proceeds to block 208 of process 200 (see Figure 2), the presence of all five (5) microsleep characteristic eye openness factor levels. If not, the sub-process 300 goes back to block 202 of process 200 (see Figure 2). It is to be understood that the number of eye openness factors used to detect the presence of each microsleep characteristic eye openness factor level may vary, for example according to the image sampling frequency and are meant as illustrative examples only.

It is to be understood that the memory associated with the microprocessor as described above, contains a plurality of reference eye closure patterns stored therein. The graphs as illustrated in Figure 5 are compared to the reference eye closure patterns. Once a match is found, a microsleep event is verified and the alarm is activated It is to be understood that the memory associated with the microprocessor as described above, contains a plurality of reference eye closure patterns stored therein. The graphs as illustrated in Figure 5 are compared to the reference eye closure patterns. Once a match is found, a microsleep event is verified and the alarm is activated

Although the present invention has been described by way of a particular embodiment and examples thereof, it should be noted that it will be apparent to Although the present invention has been described by way of a particular embodiment and examples thereof, it should be noted that it will be apparent to

persons skilled in the art that modifications may be applied to the present particular embodiment without departing from the scope of the present invention. persons skilled in the art that modifications may be applied to the present particular embodiment without departing from the scope of the present invention