Beyond the 3 million are a significant number of women and men (teens and children as well) who are classified as "sympathizers" or "nightstand" Buddhists. They are likely not members of traditional Buddhist Temples but practice meditation, meet with meditation groups, read and study Buddhist writings (hence the label "nightstand" Buddhists). The figure for this group is not easily available but could be anywhere between 2 and 10 million. We'll us 5 million as mid point figure.
Along with the sympathizers there are those who say that Buddhism is important in shaping them in ways spiritual but not religious. A recent survey discovered that 12 percent of respondents made this connection. This spiritual but not religious category is growing and is estimated to be 30 million or more.
Taking all of these factors and figures into account, a more realistic statistic for the number of people who practice Buddhism in the USA is closer to 40 million.