ImageNet-A

by Dan HendrycksPublic Domain

ImageNet-A

We introduce natural adversarial examples -- real-world, unmodified, and naturally occurring examples that cause classifier accuracy to significantly degrade. We curate 7,500 natural adversarial examples and release them in an ImageNet classifier test set that we call ImageNet-A. This dataset serves as a new way to measure classifier robustness. Like l_p adversarial examples, ImageNet-A examples successfully transfer to unseen or black-box classifiers. For example, on ImageNet-A a DenseNet-121 obtains around 2% accuracy, an accuracy drop of approximately 90%. Recovering this accuracy is not simple because ImageNet-A examples exploit deep flaws in current classifiers including their over-reliance on color, texture, and background cues. We observe that popular training techniques for improving robustness have little effect, but we show that some architectural changes can enhance robustness to natural adversarial examples. Future research is required to enable robust generalization to this hard ImageNet test set.

Dataset Attributes

Label SVG
TasksAdversarial Examples
Label SVG
CategoriesNatural Image
Label SVG
SensorWebsampling

Class Labels

StingrayGoldfinch; Carduelis CarduelisJunco; SnowbirdRobin; American Robin; Turdus MigratoriusJayBald Eagle; American Eagle; Haliaeetus LeucocephalusVultureEftBullfrog; Rana CatesbeianaBox Turtle; Box TortoiseCommon Iguana; Iguana; Iguana IguanaAgamaAfrican Chameleon; Chamaeleo ChamaeleonAmerican Alligator; Alligator MississipiensisGarter Snake; Grass SnakeHarvestman; Daddy Longlegs; Phalangium OpilioScorpionTarantulaCentipedeSulphur-Crested Cockatoo; Kakatoe Galerita; Cacatua GaleritaLorikeetHummingbirdToucanDrakeGooseKoala; Koala Bear; Kangaroo Bear; Native Bear; Phascolarctos CinereusJellyfishSea Anemone; AnemoneFlatworm; PlatyhelminthSnailCrayfish; Crawfish; Crawdad; CrawdaddyHermit CrabFlamingoAmerican Egret; Great White Heron; Egretta AlbusOystercatcher; Oyster CatcherPelicanSea LionChihuahuaGolden RetrieverRottweilerGerman Shepherd; German Shepherd Dog; German Police Dog; AlsatianPug; Pug-DogRed Fox; Vulpes VulpesPersian CatLynx; CatamountLion; King Of Beasts; Panthera LeoAmerican Black Bear; Black Bear; Ursus Americanus; Euarctos AmericanusMongooseLadybug; Ladybeetle; Lady Beetle; Ladybird; Ladybird BeetleRhinoceros BeetleWeevilFlyBeeAnt; Emmet; PismireGrasshopper; HopperWalking Stick; Walkingstick; Stick InsectCockroach; RoachMantis; MantidLeafhopperDragonfly; Darning Needle; Devils Darning Needle; Sewing Needle; Snake Feeder; Snake Doctor; Mosquito Hawk; Skeeter HawkMonarch; Monarch Butterfly; Milkweed Butterfly; Danaus PlexippusCabbage ButterflyLycaenid; Lycaenid ButterflyStarfish; Sea StarWood Rabbit; Cottontail; Cottontail RabbitPorcupine; HedgehogFox Squirrel; Eastern Fox Squirrel; Sciurus NigerMarmotBisonSkunk; Polecat; Wood PussyArmadilloBaboonCapuchin; Ringtail; Cebus CapucinusAfrican Elephant; Loxodonta AfricanaPuffer; Pufferfish; Blowfish; GlobefishAcademic Gown; Academic Robe; Judges RobeAccordion; Piano Accordion; Squeeze BoxAcoustic GuitarAirlinerAmbulanceApronBalance Beam; BeamBalloonBanjoBarnBarrow; Garden Cart; Lawn Cart; WheelbarrowBasketballBeacon; Lighthouse; Beacon Light; PharosBeakerBikini; Two-PieceBowBow Tie; Bow-Tie; BowtieBreastplate; Aegis; EgisBroomCandle; Taper; Wax LightCanoeCastleCello; VioloncelloChainChestChristmas StockingCowboy BootCradleDial Telephone; Dial PhoneDigital ClockDoormat; Welcome MatDrumstickDumbbellEnvelopeFeather Boa; BoaFlagpole; FlagstaffForkliftFountainGarbage Truck; DustcartGobletGo-KartGolfcart; Golf CartGrand Piano; GrandHand Blower; Blow Dryer; Blow Drier; Hair Dryer; Hair DrierIron; Smoothing IronJack-O-LanternJeep; LandroverKimonoLighter; Light; Igniter; IgnitorLimousine; LimoManhole CoverMaracaMarimba; XylophoneMaskMittenMosqueNailObeliskOcarina; Sweet PotatoOrgan; Pipe OrganParachute; ChuteParking MeterPiggy Bank; Penny BankPool Table; Billiard Table; Snooker TablePuck; Hockey PuckQuill; Quill PenRacket; RacquetReelRevolver; Six-Gun; Six-ShooterRocking Chair; RockerRugby BallSaltshaker; Salt ShakerSandalSax; SaxophoneSchool BusSchoonerSewing MachineShovelSleeping BagSnowmobileSnowplow; SnowploughSoap DispenserSpatulaSpider Web; Spiders WebSteam LocomotiveStethoscopeStudio Couch; Day BedSubmarine; Pigboat; Sub; U-BoatSundialSuspension BridgeSyringeTank; Army Tank; Armored Combat Vehicle; Armoured Combat VehicleTeddy; Teddy BearToasterTorchTricycle; Trike; VelocipedeUmbrellaUnicycle; MonocycleViaductVolleyballWasher; Automatic Washer; Washing MachineWater TowerWine BottleWreckGuacamolePretzelCheeseburgerHotdog; Hot Dog; Red HotBroccoliCucumber; CukeBell PepperMushroomLemonBananaCustard ApplePomegranateCarbonaraBubbleCliff; Drop; Drop-OffVolcanoBallplayer; Baseball PlayerRapeseedYellow Ladys Slipper; Yellow Lady-Slipper; Cypripedium Calceolus; Cypripedium ParviflorumCornAcorn