Competitors investigate Madoff’s impossible numbers and alert the Securities and Exchange Commission, but the agency shrugs off multiple red flags.