If the total number of open orders exceeds 10,000 then any new orders will be rejected. If an account wishes to open more than 10,000 orders then the account holder may consider optimizing the order efficiency on the account. For more on order efficiency optimization please visit the IBKR Knowledge Base.
However, there are restrictions on the account level to the number of orders one can have. For example, each time an order is entered, the account's credit is checked (the impact on excess liquidity on all open orders is added up). When the excess liquidity limit is reached, new opening orders will be rejected.