Although many generations have been born into freedom, these animals cannot be considered truly wild, but feral. Feral is defined as descending from domesticated animals, and these horses are the descendants of domestic stock that had been selectively bred for thousands of years on both sides of the Atlantic. Although horses originally evolved on the North American continent, native wild horses became extinct here about 10,000 years before the first European explorer arrived. Equine geneticists believe that the Banker Ponies descend from Spanish stock, most probably from the Spanish herds established in the West Indies beginning in the early 1500s.

With the discovery of the New World, Spanish conquistadors brought mounts from their homeland to carry them on their explorations and into battle with the native people. Trans-Atlantic voyages were long, and up to half of the horses transported succumbed in transit. Traveling with livestock was unpleasant, and large stores of food and water were necessary to sustain them. The Spaniards recognized the need to establish breeding ranches, so that a more convenient, practical source of good mounts might be readily available to explorers and others. Columbus brought twenty-five horses on his second voyage in 1493, and by 1500 successful ranches had been established in Puerto Rico, Santo Domingo, and Cuba. The foundation stock was of finest quality and commanded high prices. The conquistadors paid in gold, which the breeders re-invested in more fine breeding stock to keep up with the demand.

These Spanish horses were known as Jennets, or Jenettas. Originally bred as war horses, they were respected world-wide for their intelligence, stamina, power, and beauty. The Jennet was somewhat short in stature, distinctively short-legged and deep through the chest, with a roman nose, sloping rump, short back, and luxurious mane and tail. While most horses have six lumbar vertebrae, or a fusion of the fifth and sixth, Spanish horses tended to have only five, a trait shared with Arabians and a few other breeds, , although individuals with five vertebrae can be found in almost every breed. While the original Jennet does not exist today, Jennet bloodlines evolved into modern breeds such as the Andalusian, Paso Fino, and Lipizzaner.

When the English defeated the Spanish Armada in 1588, North America became more open for English colonization. The new settlements required livestock, but faced the same importation problems that had beset the Spanish. Many settlers chose, instead, to follow the example set by the Grenville expedition in 1585: they purchased their animals from the large breeding ranches in the West Indies, thereby contending with livestock only on the final, easier leg of the journey. In this way, English colonies acquired Spanish cattle with long, curving horns; Spanish sheep, goats, and swine; and Spanish horses.

Some livestock may have arrived on the Outer Banks incidentally. When the Tiger ran aground in 1585, probably in Ocracoke Inlet, the livestock aboard may have been unloaded to lighten the ship. If Sir Francis Drake brought livestock with him on his visit in 1586, some may have come ashore when a storm scattered his fleet. Any of several other English missions to what is now North Carolina could have left domestic animals on the barrier chain by mishap. In all likelihood, however, the introduction of horses and other livestock was deliberate. The most significant introductions probably didn't begin until the second half of the seventeenth century, when white people began filtering into the area from increasingly crowded settlements on Chesapeake Bay and its tributaries. And most of the accidental introductions were probably made by inbound Englishmen, not by outbound Spaniards. (Having imported horses in a campaign to conquer the New World and exploit its wealth, why would the Spanish have left gold and silver on the dock in order to take horses back to Spain?)

Coastal settlers quickly learned to use barrier islands as fenceless pastures. The animals were contained by sea and sound, and the marsh grass kept them well-fed. There were no permanent Indian settlements and few large predators. Pennings were held once or twice a year to brand young stock or capture animals for use on the mainland. At most other times, the animals fended for themselves.

Eventually, people began to settle the islands, but by then feral livestock roamed in abundance.Villagers built fences around their property to keep the animals out of their gardens, and put wooden latticework beneath their homes to prevent insect-infested animals from seeking shade there. Horse populations grew rapidly. In the 1920s, National Geographic documented 2000–3000 horses roaming on the Outer Banks. As the human population grew around the Nags Head area, which became a resort for mainland planters in the early 1800s, the feral horses moved to the north and south of civilization. The islands themselves were constantly dividing and reuniting, sculpted by the action of sea, wind, and storms. For a span of ninety-one years prior to 1846, when a storm carved Hatteras inlet, it was possible to walk from Hatteras to Ocracoke. Horses would migrate from one island to the next, become separated by inlets, then rejoin the other groups as sediment refilled them or by swimming to the opposite shore.

By the onset of the Depression, the Outer Banks had been in economic decline for a generation. Many areas had never recovered from the Civil War. Bridges built with county and private funds opened the northern Banks for development in the early 1930s, and a State highway from Kitty Hawk to Nags Head completed in 1932 sealed the fate of this area. Developers wanted an artificial dune line east of the new highway to protect it, the new construction that they hoped it would bring, and the existing villages from storm surge and drifting sand. The federal and State governments built this defense at considerable cost and planted a great deal of vegetation to stabilize it. Roaming livestock would have removed the anchoring vegetation in short order. So beginning in 1937, free range disappeared from the Outer Banks.

Most Outer Banks ranchers didn't own large tracts of property, but made a living (or part of a living) grazing cattle and other animals at liberty on common land. Under the new laws, most couldn't continue an occupation that had been pursued by generations of Outer Bankers.

Livestock continued to roam free on Ocracoke until the 1950s. During this decade the island earned the distinction of being the home of the only mounted Boy Scout troop in North Carolina if not in the world. Each Ocracoke Scout would select, capture, and train a wild-living Banker pony, usually a stallion. Under the guidance of Capt. Marvin Howard, the troop successfully competed against purebred horses in local races and displayed skilled horsemanship in pony penning festivals and parades. N.C. Highway 12 was paved piecemeal through the 1940s and 1950s, putting the free-roaming livestock at risk. Cattle were removed from the island, but some horses were permitted to remain in a small fenced-off area to the west, where they may be visited today.