If it is considered toxic then it is poison regardless the amount.
Actually, the physician Paracelsus in the 15th century was noted for the observation “It’s the dose that makes the poison.” The full quote is something like “All things are poison, and nothing is without poison; only the dose permits something not to be poisonous.” I also read a similar quote by a toxicologist: “There is no such thing as toxic substances, only toxic quantities.”
This is still the prevailing wisdom; in sufficient quantities, everything is toxic, including water (Google “water poisoning”). In small enough quantities, nothing will hurt you. Our water quality is determined by how concentrated the pollutants are, not whether they exist or not. They always exist. For instance, the EPA has set a limit of water contamination by dioxin, a very toxic substance, to 0.00000003 mg/L.
I suspect the common fear of food coloring has its roots in the banning in 1976 of FD&C Red No. 2, also known as amaranth. The dye had been linked to cancer in a Russian study. This was the reason that M&M’s stopped making the red ones, even though they used a different dye.