More Data, Weaker AI Predictions: Unexpected Results From Korean Consumer Study

A Korean study found that extra information often made predictions worse under the original models tested.

Every extra question on a customer survey asks someone to give up a little more time and personal information. Businesses collecting those answers may expect a clearer picture of who will use services such as AI shopping assistance and recommendations. A Korean study found that extra information often made predictions worse under the original models tested.

Researchers Soonho Jang and Eung-Kyo Suh explored this issue in "Questionnaire Item Budgets and Incremental Predictive Value for Self-Reported AI-Enabled Consumer Service Use: Temporal Evidence from a Korean Panel." Their paper appeared in the Journal of Theoretical and Applied Electronic Commerce Research. It examined whether additional survey questions improved predictions beyond what previous service use or a basic overall usage rate could provide.

The main analysis focused on people who hesitated to shop online because their activity might be recorded. These respondents offered a relevant setting for asking whether collecting more information delivered a measurable predictive benefit. The findings connected that benefit to the questions chosen, the people being studied, and the way models handled their answers.

What the researchers asked and measured

The team used the Korea Information Society Development Institute's panel surveys from 2023, 2024, and 2025. Models developed with 1,565 respondents used 2023 information to predict reported use in 2024, with later testing covering 1,520 respondents using the next annual transition. The later group included 641 people from model development and 879 people absent from that development sample.

Researchers tested allowances ranging from zero to 30 additional questions, covering demographics, expected benefits, personal-data management, expected safeguards, consent responses, generative AI experience, and digital competence. They counted the original questions people answered, meaning an eleven-question competence score still cost eleven answers despite entering a model as one measure. Previous service use was treated as existing information when available, with separate tests examining situations where demographics were already known.

The original approaches were ridge logistic regression and a tree-based method called histogram-based gradient boosting. Each selected question set was compared with a model using every candidate information block and a simple benchmark requiring no extra collection. Performance was measured mainly through log loss, a score that penalises incorrect probability predictions, with lower values indicating better performance.

More answers did not guarantee better predictions

All four original models using every candidate block produced higher prediction errors than their respective benchmarks. One model combining previous use with all additional information recorded a loss of 0.69961, compared with 0.66062 for the previous-use benchmark. Evidence supported deterioration in three of the four comparisons after accounting for the number of comparisons made.

None of the original question-budget policies established an improvement over its existing-information benchmark under the main tests. A single question about expected benefits was selected in every original one-question setting, producing estimated error reductions of roughly 0.5% to 1.2%. Those small gains remained uncertain under the main comparisons, making this question a promising candidate for further testing.

Four-question allowances consistently selected demographics, with three original demographic models showing worse later performance than their benchmarks. Repeating model development 1,000 times showed that frequently selecting the same questions did not guarantee dependable future predictions, and uncertainty ranges for all eight focal comparisons showed no difference. Models that looked useful in tests within the development period could lose that advantage when applied to the later period.

How answers were handled changed the results

Combining 17 regional categories into a capital-region indicator and re-tuning the models changed all four demographic comparisons from worse to slightly better estimated performance. The revised estimates still lacked clear evidence of improvement over the benchmarks, and the exercise changed both representation and tuning together. Stronger controls against fitting sample-specific patterns were associated with smaller excess errors, showing how model choices could influence the value of the same answers.

Adjusting predicted probabilities to better match observed outcomes recovered some lost performance without fully closing the benchmark gap in key models. Demographic-only models had little ability to distinguish users from nonusers among respondents absent from development, with performance close to chance. Broader model searches produced favourable estimates for random forests in the hesitant group without establishing a reliable full-information advantage there.

Expected benefits and digital competence made positive estimated contributions in the information-block analysis, with uncertainty remaining around both. Competence required eleven answers, giving the single benefits question a practical advantage for further questionnaire testing. Generative AI experience had the most negative estimated contribution per question across the later-period analyses, as reported experience rose from about 12% to 30% and its earlier relationship with subsequent consumer-service use weakened.

What businesses can reasonably take from the study?

Expanding the analysis to the full linked panel, with 3,895 development respondents and 3,741 evaluation respondents, produced stronger evidence of gains from extra information. Models trained on that broader panel showed clearer improvements among respondents outside the hesitant group, with much weaker support among people absent from development. These differences make the intended customer population an essential part of any decision about what to collect.

Measurement changes complicate the picture because the surveys changed service descriptions and wording across years, and the outcome captured recalled use rather than recorded behaviour. Overall reported use stayed near 56% in the restricted samples, but use among the 641 returning participants fell from about 67% to 60%, with a larger fall in monthly use. Comparisons with leisure and work services could not fully separate behavioural changes from questionnaire effects, and applying survey weights changed the direction of some model comparisons.

The study covers two annual transitions in one Korean panel, and follow-up analyses reused an evaluation period the researchers had already examined. It does not establish that shorter surveys improve privacy, fairness, trust, or customer welfare, because question counts do not directly measure those outcomes. A practical collection review should test what each question adds beyond existing information, examine how answers are represented, and confirm performance with future respondents before expanding a questionnaire.