On the cusp of a deal to acquire one of the hottest new names in makeup, directors of beleaguered beauty giant Coty Inc. had some unconventional concerns.

Will sales fall if the chief executive decides to have another child? Can a 22-year-old social-media star reliably stay out of trouble? What role will her mother play?

Last year, Coty was hatching a plan to pay $600 million for a controlling stake in the cosmetics startup founded by “Keeping Up with the Kardashians” star Kylie Jenner, a move aimed at reviving a beauty business dependent on drugstore staples like Cover Girl and Max Factor.

Coty directors ultimately decided Kylie Cosmetics’ metrics—$200 million in annual revenue with fewer than a dozen employees and virtually no ad spending—outweighed their worries. Also persuasive were reassurances from Ms. Jenner’s mother, Kris Jenner, who helped launch the company in 2015 and led negotiations.

A year after closing the deal, Kylie Cosmetics has both bolstered its new parent and dealt it some blows—though none that the company initially anticipated.

The deal gave Coty a hot brand with social-media savvy and a new skin-care line, a segment women have recently been spending more on.

When Coty announced the deal in November of 2019, executives said the plan was to meld Kylie’s wide appeal and bare-bones business structure with Coty’s reach and capabilities in manufacturing and research and development. Company executives said they planned to eventually take over production of Kylie makeup and expand the brand both geographically and into the fast-growing and lucrative skin-care category.

Ms. Jenner has a social-media following similar to Nike or Starbucks, Coty CEO Sue Nabi said in an interview. “On one side, you have Kylie Jenner knowing the next thing that people are looking for,” she said. “Then you combine that with the fantastic machinery of a company that is Coty.”

Peter Harf, Coty’s executive chairman, lauded the deal in a recent call with Wall Street analysts. “I personally think that this is a brand that could reach, I mean, hundreds of millions,” he said. “It could very, very soon be the biggest brand we have.”

Through a spokeswoman, Kylie Jenner and her “momager”—or mom-manager—Kris Jenner, both declined to comment.

But fallout from the coronavirus pandemic hobbled Kylie Cosmetics’ production this summer while giving birth to trends that analysts say could hurt the brand’s sales.

Without disclosing figures, Coty executives have said sales of Kylie Cosmetics were depressed this year by Covid-related supply-chain disruptions. The company said sales of Kylie Skin—which launched in early 2019 and includes face moisturizers, body scrubs and serums—tripled in the quarter than ended this October, compared with the same period a year earlier. Coty said Kylie Skin reached $25 million in sales in 2019.

A third-party estimate indicates that excluding Kylie Skin and in-store sales, revenue fell significantly in 2020.

The acquisition also faces another challenge: a lawsuit filed in June by the private-label manufacturer that makes and distributes Kylie Cosmetics products. Oxnard, Calif.-based Seed Beauty LLC claims in the suit that it played a key role in fueling the company’s early success and alleges that Ms. Jenner shared trade secrets with Coty as part of the deal.

Coty, whose brands also include Clairol hair dye, OPI nail polish and a stable of luxury fragrances, foundered since acquiring dozens of beauty brands from Procter & Gamble Co. in 2016. Coty stock has lost 70% of its value since that deal, and the company last year took $4 billion in write-downs on the P&> business.

Kylie Cosmetics came on the company’s radar as it sought an acquisition that would provide growth potential and entry into the world of influencer-fueled, direct-to-consumer beauty.

In talks with Coty, Kris Jenner played the role of deal maker, working closely with Mr. Harf, the company’s chairman. He is one of two managing partners of JAB Holding Co., Coty’s largest shareholder.

Coty said last January it was installing Christoph Honnefelder, an executive with European specialty retailer Douglas, as Kylie Cosmetics’ CEO. He resigned from the job in June, Coty said, citing personal reasons. Mr. Honnefelder declined to comment.

Instead, Coty installed its head of luxury brands to run Kylie Cosmetics. As it continued to build up the newly acquired unit, Coty also raised its bet on the Kardashian-Jenner clan. In June, Coty said it would pay $200 million for about 20% of the makeup brand owned by Ms. Jenner’s reality-star sister, Kim Kardashian West, KKW Holdings LLC.

Ms. Jenner was 18 when she started Kylie Cosmetics, selling nude lip liners and lipsticks promising the look of fuller lips. She had previously acknowledged using fillers to augment her lips amid controversy over her look. Kylie Cosmetics’ first products were $29 “lip kits,” which pair a pencil and matte liquid lipstick to make lips look fuller.

Ms. Jenner amassed a legion of loyal customers such as 24-year-old Kristi Murguly, who says she owns between $8,000 and $10,000 worth of Kylie Cosmetics products. Products are sold online, in department stores and at specialty retailers such as Sephora. Kylie Cosmetics and Kylie Skin lines are pricier than most drugstore mainstays, with offerings that include $12 lip gloss and $24 face moisturizer. Pricey limited-edition bundles are popular among loyalists who, like Ms. Murguly, spend hundreds of dollars on a new release.

Carrying a premium price tag, Kylie Cosmetics products are available at higher-end retailers like Sephora and Ulta.

Carrying a premium price tag, Kylie Cosmetics products are available at higher-end retailers like Sephora and Ulta.

Photo: David Dee Delgado/Getty Images

“I feel like she really put a lot of energy into the brand,” said Ms. Murguly, who enlists her friends’ help in snapping up limited-edition products before they sell out.

A typical post on Ms. Jenner’s Instagram page generates a million or more likes within hours, mostly from gushing fans. Last month she announced a $150, special-edition set of miniature-sized bath products—a bubble bath, body scrub, bath salt, and two candles—that came with an Advent calendar.

By 2016, Kylie Cosmetics had sales of $370 million, according to a person familiar with the company’s finances. The privately held company didn’t disclose sales figures for the following three years, during which time many analysts and industry watchers assumed the growth trajectory continued. But Coty, when it announced its deal with Kylie Cosmetics, disclosed 2018 sales of $125 million, roughly one-third of the 2016 figure. The company said the sales, including the skincare business, reached $200 million in 2019.

Coty declined to comment on sales in that period. It said in a statement that Coty provides a “massive global distribution network and the resources to enter new beauty categories,” to enable the startup to grow.

Online sales of Kylie Cosmetics, not including the Kylie Skin business, fell 39% through October compared with the same period in 2019, according to Rakuten Intelligence, which tracks electronic receipts. Full-year 2019 sales fell 21% from the previous year, according to the firm. Those figures don’t include sales at beauty retailer Ulta, which began selling the brand in 2019, or other physical stores.

SHARE YOUR THOUGHTS

What appeal do you think celebrity brands hold for typical consumers? Join the conversation below.

Even though the coronavirus pandemic has pushed people to do more of their shopping online, a shift beneficial to companies like Kylie Cosmetics, Americans are buying more of their beauty products at the grocery store and reverting to familiar brands, such as long-struggling Cover Girl, also owned by Coty, said Nielsen analyst Nicole Collida.

“Consumers are not necessarily engaging in new direct-to-consumer brands as much as they are engaging with mass brands,” she said.

The Seed Beauty lawsuit, filed in Los Angeles Superior Court alleging misappropriation of trade secrets and breach of contract, hasn’t stopped Coty’s plans.

The suit reflects the changing nature of contemporary beauty brands. Historically, big beauty companies have been in charge of their own manufacturing, formulations and distribution, while relying on advertising agencies to market their products and retailers to sell them. But a generation of beauty startups, including Kylie Cosmetics, is upending that formula, selling directly to consumers and relying on social media for marketing, while farming out production and distribution to third parties.

The suit casts Seed Beauty as the manufacturing and strategic muscle behind Kylie’s ascendency. Seed alleges that Ms. Jenner and Ms. Kardashian West, with KKW Beauty, shared trade secrets with Coty. The details of the secrets are redacted in the suit, which is seeking unspecified damages. Seed Beauty will “continue to protect its rights and looks forward to its day in court,” the company said in a statement this week.

Coty and Ms. Jenner’s spokeswoman declined to comment on the suit.

Alessandro Cardoso, a 24-year-old TikTok influencer, recently generated more than 81,000 likes for a video in which he included Kylie Cosmetics lip kits in a series titled “Overhyped makeup that actually sucks.”

He said he was intrigued by buzz around the brand’s famous lip kits and decided to try one out.

“I was pretty disappointed,” he said. “It was a formula that you could honestly just buy at the drugstore.”

—Renata Geraldo contributed to this article.

Write to Sharon Terlep at sharon.terlep@wsj.com

Corrections & Amplifications
Coty Inc. said sales of Kylie Skin reached $25 million in 2019 and sales of Kylie Cosmetics, including the skincare business, reached $200 million in 2019. An earlier version of this article incorrectly said sales of Kylie Skin reached $25 billion and sales of Kylie Cosmetics reached $200 billion. (Corrected on Jan. 8)