999445 - Decrease in number of users getting OpenGL layers on OS X (Intel Graphics, OSX 10.6 and 10.7, Firefox 27.0+)

Status: VERIFIED FIXED

(Core :: Graphics, defect)

Description

[Jeff Muizelaar [:jrmuizel]]

http://people.mozilla.org/~bjacob/gfx_features_stats/#mac shows a large decrease in the number of users getting layers on OS X. This seems bad.

Comment 1

[Benoit Jacob [:bjacob] (mostly away)]

Wasn't the 100% rate that we used to measure unplausible in the first place? I thought of that as a bug in how we record layers initialization success on Mac (basically, if we blacklist based on OSX version before we even try, so we only record successes) and was sort of pleased to see it drop to now realistic, and still quite high, levels.

Comment 2

[Benoit Girard (:BenWa)]

We only support 10.6 and later. We blacklist layers only on some 10.6 that are behind on their system updates which should be very minimal. I think the current drop is really unrealistic.

Comment 3

[Benoit Jacob [:bjacob] (mostly away)]

All Intel Graphics users, concentrated on only two device IDs:


bjacob:~/crash-stats$ zcat 20140416-pub-crashdata.csv.gz | grep 'Mac OS X' | grep 'Layers-' | wc -l
584
bjacob:~/crash-stats$ zcat 20140416-pub-crashdata.csv.gz | grep 'Mac OS X' | grep 'Layers-' | grep 'AdapterVendorID: 0x8086' | wc -l
582
bjacob:~/crash-stats$ zcat 20140416-pub-crashdata.csv.gz | grep 'Mac OS X' | grep 'Layers-' | grep 'AdapterVendorID: 0x8086' | sed 's/.*\(AdapterDeviceID: 0x[a-fA-F0-9]*\).*/\1/g' | sort | uniq -c 
    455 AdapterDeviceID: 0x27a2
    127 AdapterDeviceID: 0x2a02

Comment 4

[Benoit Jacob [:bjacob] (mostly away)]

Looked up these two device IDs in GfxDriverInfo.cpp;

0x27a2 is the Intel 945 (from the GMA 950 family) from a very, very long time ago (no hardware vertex shaders...)

0x2a02 is the Intel GL960 (from the GMA X3000 family) from circa 2006 or 2007, an actual hardware implementation of all of OpenGL 2 AFAIK, so that's more concerning IMHO.

The next thing is OSX versions. It's 10.6 and 10.7 exclusively:

bjacob:~/crash-stats$ zcat 20140416-pub-crashdata.csv.gz | grep 'Mac OS X' | grep 'Layers-' | grep 'AdapterVendorID: 0x8086' | sed 's/.*\(Mac OS X.[0-9.]*\).*/\1/g' | sort | uniq -c | sort -rn
    446 Mac OS X	10.6.8
    130 Mac OS X	10.7.5
      1 Mac OS X	10.7.4
      1 Mac OS X	10.7.0
      1 Mac OS X	10.6.7
      1 Mac OS X	10.6.6
      1 Mac OS X	10.6.4
      1 Mac OS X	10.6.3

Comment 5

[Benoit Jacob [:bjacob] (mostly away)]

In conclusion, this is entirely made of Intel Graphics on OSX 10.6 and 10.7. This smells like something about being low-spec in some kind of way...

Comment 6

[Benoit Jacob [:bjacob] (mostly away)]

Regarding Firefox versions, the thing of note is that (within noise) this only started appearing significantly during February with Firefox 27, and become much more prevalent during the first half of April on Firefox 28.

bjacob:~/crash-stats$ zcat 20140201-pub-crashdata.csv.gz | grep 'Mac OS X' | grep 'Layers-' | grep 'AdapterVendorID: 0x8086' | sed 's/.*\(Firefox.[0-9.]*\).*/\1/g' | sort | uniq -c | sort -rn
      5 Firefox	27.0
      3 Firefox	28.0
bjacob:~/crash-stats$ zcat 20140215-pub-crashdata.csv.gz | grep 'Mac OS X' | grep 'Layers-' | grep 'AdapterVendorID: 0x8086' | sed 's/.*\(Firefox.[0-9.]*\).*/\1/g' | sort | uniq -c | sort -rn
     24 Firefox	28.0
     22 Firefox	27.0.1
     14 Firefox	27.0
      2 Firefox	29.0
      1 Firefox	26.0
bjacob:~/crash-stats$ zcat 20140301-pub-crashdata.csv.gz | grep 'Mac OS X' | grep 'Layers-' | grep 'AdapterVendorID: 0x8086' | sed 's/.*\(Firefox.[0-9.]*\).*/\1/g' | sort | uniq -c | sort -rn
     66 Firefox	27.0.1
     10 Firefox	28.0
      2 Firefox	27.0
bjacob:~/crash-stats$ zcat 20140315-pub-crashdata.csv.gz | grep 'Mac OS X' | grep 'Layers-' | grep 'AdapterVendorID: 0x8086' | sed 's/.*\(Firefox.[0-9.]*\).*/\1/g' | sort | uniq -c | sort -rn
     69 Firefox	27.0.1
     12 Firefox	28.0
      2 Firefox	29.0
      1 Firefox	27.0
bjacob:~/crash-stats$ zcat 20140401-pub-crashdata.csv.gz | grep 'Mac OS X' | grep 'Layers-' | grep 'AdapterVendorID: 0x8086' | sed 's/.*\(Firefox.[0-9.]*\).*/\1/g' | sort | uniq -c | sort -rn
     41 Firefox	28.0
     18 Firefox	27.0.1
     10 Firefox	29.0
      2 Firefox 
      1 Firefox	27.0
bjacob:~/crash-stats$ zcat 20140405-pub-crashdata.csv.gz | grep 'Mac OS X' | grep 'Layers-' | grep 'AdapterVendorID: 0x8086' | sed 's/.*\(Firefox.[0-9.]*\).*/\1/g' | sort | uniq -c | sort -rn
     65 Firefox	28.0
     10 Firefox	29.0
      5 Firefox	27.0.1
      2 Firefox	30.0
      1 Firefox	27.0
      1 Firefox	26.0
bjacob:~/crash-stats$ zcat 20140410-pub-crashdata.csv.gz | grep 'Mac OS X' | grep 'Layers-' | grep 'AdapterVendorID: 0x8086' | sed 's/.*\(Firefox.[0-9.]*\).*/\1/g' | sort | uniq -c | sort -rn
    505 Firefox	28.0
    101 Firefox	29.0
     27 Firefox	27.0.1
     12 Firefox	27.0
      9 Firefox	30.0
      1 Firefox	31.0
      1 Firefox	25.0.1
      1 Firefox	24.0
      1 Firefox	21.0
      1 Firefox 
bjacob:~/crash-stats$ zcat 20140415-pub-crashdata.csv.gz | grep 'Mac OS X' | grep 'Layers-' | grep 'AdapterVendorID: 0x8086' | sed 's/.*\(Firefox.[0-9.]*\).*/\1/g' | sort | uniq -c | sort -rn
    718 Firefox	28.0
    113 Firefox	29.0
     21 Firefox	27.0.1
     12 Firefox	30.0
      5 Firefox	27.0
      2 Firefox 
      1 Firefox	31.0
      1 Firefox	12
      1 Firefox!

Comment 7

[Benoit Jacob [:bjacob] (mostly away)]

Hey, I updated again my stats, and we're most of the way back up now!

http://people.mozilla.org/~bjacob/gfx_features_stats/#mac

Still not quite where we used to be. Let's keep this bug open for at least a few weeks and monitor this graph (which I have to update manually, so ping me).

Comment 8

[Benoit Jacob [:bjacob] (mostly away)]

And on http://people.mozilla.org/~bjacob/gfx_features_stats/#os-market-share this seems correlated with a OSX crashiness spike...

Comment 9

[Benoit Jacob [:bjacob] (mostly away)]

If someone feels like taking a look, here are 20 random crash links from affected users on April 11:

bjacob:~/crash-stats$ zcat 20140411-pub-crashdata.csv.gz | grep 'Mac OS X' | grep 'Layers-' | cut -d '       ' -f 3 | head -n20
http://crash-stats.mozilla.com/report/index/0526cae5-7ede-4236-8813-6f6262140411
http://crash-stats.mozilla.com/report/index/5134b935-43d5-4371-a22f-d30da2140411
http://crash-stats.mozilla.com/report/index/f8c1cef7-f512-4e45-9deb-b12d42140411
http://crash-stats.mozilla.com/report/index/1a624d36-02f4-4069-aa9d-955662140411
http://crash-stats.mozilla.com/report/index/bfda4008-abe1-48c9-a1c7-3bb2d2140411
http://crash-stats.mozilla.com/report/index/bbe944d8-6490-473e-848a-97ecf2140411
http://crash-stats.mozilla.com/report/index/8dc7f56a-78e8-4393-b92b-b56892140411
http://crash-stats.mozilla.com/report/index/c6282504-22d2-4021-ba96-77f1e2140411
http://crash-stats.mozilla.com/report/index/d97d12b0-8f91-4f1d-bf6d-ad20a2140411
http://crash-stats.mozilla.com/report/index/b571e169-958d-40c5-a785-ae6162140411
http://crash-stats.mozilla.com/report/index/0eb52689-3e19-4617-a2e1-ed1ee2140411
http://crash-stats.mozilla.com/report/index/837263b7-b617-46dc-b1e7-0b11a2140411
http://crash-stats.mozilla.com/report/index/9e88677f-98c3-446d-a5ee-7e2ab2140411
http://crash-stats.mozilla.com/report/index/2d8b866e-ceac-4d8a-87f7-26b092140411
http://crash-stats.mozilla.com/report/index/80462ee7-8b39-4e57-8e41-5859c2140411
http://crash-stats.mozilla.com/report/index/681209be-2195-4b8d-ba0d-13d4c2140411
http://crash-stats.mozilla.com/report/index/5fb20eca-4220-4af6-8752-91d302140411
http://crash-stats.mozilla.com/report/index/be1a06da-f91b-4799-bc7a-3baab2140411
http://crash-stats.mozilla.com/report/index/8ce06851-3126-4fd4-9035-77b212140411
http://crash-stats.mozilla.com/report/index/46324d0b-7bf0-4dce-9d8f-08a902140411

Comment 10

[Benoit Jacob [:bjacob] (mostly away)]

Over 80% of these are flash plugin crashes; here's another run filtering only non-plugin crashes:

bjacob:~/crash-stats$ zcat 20140411-pub-crashdata.csv.gz | grep 'Mac OS X' | grep 'Layers-' | grep -v plugin | cut -d '      ' -f 3 | head -n20
http://crash-stats.mozilla.com/report/index/5134b935-43d5-4371-a22f-d30da2140411
http://crash-stats.mozilla.com/report/index/8ce06851-3126-4fd4-9035-77b212140411
http://crash-stats.mozilla.com/report/index/46324d0b-7bf0-4dce-9d8f-08a902140411
http://crash-stats.mozilla.com/report/index/89075571-ad97-42f3-8e5b-fc0dd2140411
http://crash-stats.mozilla.com/report/index/c0830413-10d4-4a77-8c35-4518d2140411
http://crash-stats.mozilla.com/report/index/894aeb50-051d-47df-a2ae-de3772140411
http://crash-stats.mozilla.com/report/index/e10e008c-8873-4dd9-bda4-956062140411
http://crash-stats.mozilla.com/report/index/59ec294b-dd4f-47c8-921e-0345a2140411
http://crash-stats.mozilla.com/report/index/327e388f-14e7-4401-9ee8-3f06a2140411
http://crash-stats.mozilla.com/report/index/92c2923a-67ad-49c5-9c44-5762c2140411
http://crash-stats.mozilla.com/report/index/94352fdd-e720-4500-8276-fedd52140411
http://crash-stats.mozilla.com/report/index/637daac9-f88c-487e-a9ff-c99ed2140411
http://crash-stats.mozilla.com/report/index/9268adfb-3274-419e-a9e4-ab3ae2140411
http://crash-stats.mozilla.com/report/index/b610959e-79bd-495b-bcbd-ddc302140411
http://crash-stats.mozilla.com/report/index/05408d72-3bd4-40ad-9167-63fc42140411
http://crash-stats.mozilla.com/report/index/cbf4faa9-2732-4a19-a211-809482140411
http://crash-stats.mozilla.com/report/index/7f93fe00-8045-44b8-b26b-224072140411
http://crash-stats.mozilla.com/report/index/4a3504af-2fb5-4957-910c-1b4e22140411
http://crash-stats.mozilla.com/report/index/ec7b67aa-a825-450e-9547-c8d612140411
http://crash-stats.mozilla.com/report/index/0b20156f-7b7f-48ef-bd21-25d702140411

Comment 11

[Benoit Jacob [:bjacob] (mostly away)]

Updated http://people.mozilla.org/~bjacob/gfx_features_stats/ .

Mac GL layers rates are still not back up to 100%. They're sort of plateauing around 95%.

Comment 12

[Milan Sreckovic [:milan] (needinfo for best results)]

It's good that we're tracking this and that we may get some telemetry.  There may be a number of different explanations for this, given that we're picking up the data from crash reports and that the crash reports spiked at the same time.

Comment 13

[Milan Sreckovic [:milan] (needinfo for best results)]

The timing and versions may not quite match, but can't help but wonder if bug 1004134 (thanks for a CC :gal), is related to this (10.9, intel graphics)

Comment 14

[Benoit Jacob [:bjacob] (mostly away)]

Attachment: Table showing corresponding decrease of WebGL success rates

Because people have also noticed (and seem to be anxiously tracking) the corresponding decrease of the WebGL success rate on the public_webgl list, I looked into that. It's also concentrated on OSX 10.6 and 10.7 only, and is already back to normal on 10.7, so only 10.6 remains affected.

Comment 15

[Benoit Jacob [:bjacob] (mostly away)]

Note: generated by

for DATE in 20140101 20140115 20140201 20140215 20140301 20140315 20140401 20140415 20140420 20140425 20140428 20140501 20140503; do echo "<tr>"; echo "<td>$DATE</td>"; for VERSION in 10.6 10.7 10.8 10.9; do echo "<td> $(echo "100 * `zcat $DATE-pub-crashdata.csv.gz | grep Mac.OS.X.$VERSION | grep 'WebGL+' | wc -l` / `zcat $DATE-pub-crashdata.csv.gz | grep Mac.OS.X.$VERSION | grep 'WebGL[?!]' | wc -l`" | bc) % </td>"; done; echo "</tr>"; done

Comment 16

[Jeff Muizelaar [:jrmuizel]]

This seems to have been caused by bug 923194. Benoit will create a fix.

Comment 17

[Benoit Jacob [:bjacob] (mostly away)]

Many thanks Jeff :)

(The conciseness of comment 16 might not make this clear, but Jeff just did a careful bisection on a Mac Mini that reproduces.)

To be clear, what happened is that in addition to doing what it says it does, bug 923194 also made us actually refuse to initialize GL layers on too-low GL versions. The affected Macs have GL 1.4 so they are rejected. They should not be rejected, and they used to work fine with GL layers, because they support all the GL extensions that we need. Basically, we need to do a proper "GL feature" check like we already do for various things. Patch coming.

Comment 18

[Benoit Jacob [:bjacob] (mostly away)]

Attachment: dont-require-gl-version

Comment 19

[Benoit Jacob [:bjacob] (mostly away)]

Filed bug 1006050 to do the right thing i.e. properly verify that the GL supports the features that we need, on initialization. (I'm not planning to work on it at this point).

For now, the patch here is minimal to make it easy to uplift to beta.

Comment 20

[Benoit Jacob [:bjacob] (mostly away)]

https://hg.mozilla.org/integration/mozilla-inbound/rev/ee3f79376eef

Comment 21

[Benoit Jacob [:bjacob] (mostly away)]

Comment on attachment 8417519 [details] [diff] [review]
dont-require-gl-version

[Approval Request Comment]
Bug caused by (feature/regressing bug #): bug 923194 / Firefox 27
User impact if declined: About 20% of Mac OS 10.6 users (5% of total Mac users) get no hardware acceleration (so they have a slower and far less tested code path) and also no WebGL.
Testing completed (on m-c, etc.): m-i
Risk to taking this patch (and alternatives if risky): Should not be risky, since on release builds it's just reverting bug 923194. Basically, I think it's a no-brainer for aurora; regarding beta, well this is clearly a nontrivial switch for beta, but my perspective here is that it is even more scary to me (as a gfx dev) to have 5% of release channel users on a code path (non-GL layers) that we didn't even know was still functional on Mac. So I'd like this to be fixed asap, which is why I would still like this to be taken on beta. Hopefully it's early enough in the cycle for that.
String or IDL/UUID changes made by this patch: none

Comment 22

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

https://hg.mozilla.org/mozilla-central/rev/ee3f79376eef

Comment 23

[Sylvestre Ledru [:Sylvestre]]

Comment on attachment 8417519 [details] [diff] [review]
dont-require-gl-version

Early in the beta process, we can accept it.

Comment 24

[Benoit Jacob [:bjacob] (mostly away)]

https://hg.mozilla.org/releases/mozilla-aurora/rev/22d65607166f
https://hg.mozilla.org/releases/mozilla-beta/rev/f15b4b99adc4

Comment 25

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-b2g30_v1_4/rev/f15b4b99adc4

Comment 26

[Benoit Jacob [:bjacob] (mostly away)]

(In reply to Ryan VanderMeulen [:RyanVM UTC-4] from comment #25)
> https://hg.mozilla.org/releases/mozilla-b2g30_v1_4/rev/f15b4b99adc4

Hah, OK. I should have unflagged: there is no way that OpenGL ES 2.0 platforms (i.e. all of mobile) could be affected. Contrary to desktop GL, in the ES world, 2.0 is not just 1.x plus extensions.

Comment 27

[u279076]

Looks like the downward trend has been reversed. Marking this bug verified fixed.