Leiphone.com tests DeepSeek V4.1-Flash: Is the periodic weakness found by ByteDance Seed still present?

If the same piece of code is preceded only by irrelevant tokens, will DeepSeek change its answer?

Why does DeepSeek sometimes answer correctly and sometimes incorrectly? For the same code completion problem, if only a few irrelevant tokens are added in front, DeepSeek may change its answer. (For details, see: ByteDance discovers a hidden bug in DeepSeek: add a few spaces and the model fails.) This is a phenomenon discovered by ByteDance Seed in a study at the end of September. The researchers observed that the DeepSeek-V4-Flash base model periodically switches between the correct answer "8" and the incorrect answer "32" as the length of irrelevant comments before the code changes. (Leiphone) But can this anomaly, which existed in the old base model, still be measured today? We took the code used in the Seed paper and, through DeepSeek's official API, conducted two rounds of testing on the currently deployed V4.1-Flash. The results were not the same as the typical case in the paper: all 64 groups with fixed sampling answered correctly, and in the other round of testing the correct answer also always occupied the top predicted probability, but the probability it received changed with the input.

02

ByteDance found DeepSeek's periodic weakness. To reduce the inference cost of long contexts, DeepSeek V4 adopted two alternating attention mechanisms. Schematic diagram of DeepSeek V4's CSA and HCA architectures.

CSA (compressed sparse attention) groups 8 tokens into one compression window, generates one compression record every 4 tokens, and then filters out the information most relevant to the current question.

HCA (heavy compressed attention) compresses more aggressively, retaining only one compression record for every 128 tokens, but it reads all historical compression records. The two mechanisms do different jobs: the former preserves fine-grained retrieval capability as much as possible, while the latter provides global information at a higher compression rate. The paper found that the performance drift of the DeepSeek V4 series is related to the block KV compression of CSA and HCA. The same piece of information, merely by moving a few tokens forward or backward, may fall into a different compression position, changing the effect of retention and retrieval. The researchers call this position "phase." If the model is better at retrieving information in some phases but tends to ignore information in others, then even if the key information itself has not changed, the model may change from answering correctly to answering incorrectly. This is the so-called "phase sensitivity." Figure: schematic diagram of the 8-token compression window and 4-token stride. To verify this phenomenon, the team extended the experiment to a 128K context, having the model search for specified information among 16,000 key-value records. As a result, the best-performing position group of DeepSeek-V4-Flash-Base reached 69.9% accuracy, while the worst was only 29.7%, a difference of 40.2 percentage points, and the fluctuation period was consistent with the compression stride. Where exactly is the problem? Seed further trained multiple control models based on the Qwen3-0.6B architecture to try to rule out other factors. The researchers removed the positional encoding RoPE, and the periodic weakness still existed; they canceled the learnable compression gating and switched to uniform averaging, and the weakness did not disappear either. When the compression stride was set to 4, 6, 8, and 12 respectively, the period of accuracy fluctuation changed accordingly, while the full-attention control model did not show the same obvious phenomenon. These experiments point the problem to the block cache compression mechanism, rather than just an accidental error in a certain code completion. Of course, the 40.2 percentage points is the gap between the best and worst positions of a specific base model in a controlled retrieval experiment, and cannot be directly understood as a decline in DeepSeek's accuracy in daily use.

03

We tested DeepSeek V4.1:

The answer did not flip, but the probability fluctuated. We referred to the FP8 code completion experiment in the paper, called DeepSeek's official API, and ran a round of tests on the currently deployed V4.1-Flash. (The model in which Seed initially observed answer flipping was DeepSeek-V4-Flash-Base, which can no longer be called through the official API we used.) The test method is not complicated: keep the code unchanged, only adjust the length of the irrelevant comments in front, and have the model repeatedly complete the same problem. The correct answer is "8", and the paper once observed that the model sometimes preferred the wrong "32". To minimize the influence of other factors, we used the official inference code of the version specified in the paper and truncated it to the same completion position. After checking with the Tokenizer, the code prefix was consistent with the paper, both being 983 Tokens. Next, we added irrelevant content of different lengths in front of the code, keeping the code that actually needed to be completed unchanged. The test used four types of padding in total, with 16 lengths for each type, gradually increasing from 24 Tokens to 39 Tokens. This produced 64 groups of inputs in total, with the number of input Tokens in adjacent groups differing by only 1. Figure: Comparison of actual test inputs, organized from saved API requests; the two groups of inputs differ by only one Token. In the first round of results, DeepSeek performed quite stably. In the 64 tests with fixed sampling settings, the model completed all of them with the correct answer "8", and there was no flip between the correct answer and the wrong answer as in the Seed paper. However, the model answering correctly does not mean it has the same confidence in the answer every time. We then conducted 64 probability measurements, recording the model's prediction preference for the correct answer "8" and the wrong answer "32". As a result, "8" was always the candidate with the highest probability, but the probability it received was not fixed. In the docstring group of tests, merely increasing the front padding from 24 Tokens to 25 Tokens raised P(8) from 83.33% to 92.29%. We counted the four padding methods separately and found that in this round of testing, under odd lengths, the average predicted probability of "8" was higher than under even lengths, with a gap of 3.6 to 5.8 percentage points. However, these changes are not yet enough to show that we reproduced the periodic weakness discovered by Seed. Although all four types of padding showed the phenomenon of higher average probability under odd lengths, not every pair of adjacent inputs followed this pattern. We also repeated measurements on 8 groups of inputs and found that even with completely identical inputs, the probability the model gave to "8" fluctuated, with a maximum difference of about 5.35 percentage points. In addition, in the probability tests, the wrong answer "32" was randomly generated 6 times, but in those cases the model still considered "8" more likely; the final sampling merely selected "32", which does not mean the prediction preference reversed. Therefore, this set of actual tests did not observe the answer preference flip in the Seed paper, but it did record probability changes under different padding lengths. Considering that we tested the newer V4.1-Flash and have not yet fully ruled out the influence of padding content and API fluctuations, we cannot yet attribute this change to the cache compression architecture.

04

DeepSeek made the cache smaller,

but the positional weakness actually eased. We did not observe answer flipping in this test, but this does not mean V4.1 has completely eliminated this problem. The Seed paper also found periodic prediction preference changes under some test conditions for V4.1. Different input settings and model calling methods may all affect the final result. However, judging from the long-context retrieval experiment, DeepSeek's new version has indeed shown obvious improvement. The Seed paper shows that for the V4-Flash base model, the accuracy difference between the best and worst phases was 40.2 percentage points, which narrowed to 19.1 percentage points after post-training; by the V4.1-Flash released in September, this gap further dropped to 6.1 percentage points. It is worth noting that V4.1 did not abandon cache compression; instead, it further improved compression efficiency. DeepSeek reduced the stride of some compression paths from 4 Tokens to 2 Tokens, and adopted a new Causal Encoder–Decoder (CED) architecture, combining technologies such as cross-layer cache reuse and low-precision KV storage to continue reducing costs. According to DeepSeek's official data, compared with the previous generation, V4.1's KV Cache memory requirement dropped to one quarter, and the persistent cache storage requirement dropped to one eighth. Figure 2 | Accuracy differences between different versions at the best and worst phases (chart based on data from the Seed paper). ByteDance Seed itself has also been studying long-context efficiency. In February 2025, the FlexPrefill research involving Seed attempted to dynamically select the attention positions that need to be computed based on input content; in September this year, another PISA study proposed reducing the computational overhead of sparse attention through hierarchical retrieval. It is worth noting that one of the authors of FlexPrefill, Yiyuan Ma, is also the corresponding author of the paper that discovered DeepSeek's periodic weakness.

05

Why can average scores mask the local weaknesses of new architectures? In our 64 groups of probability tests, DeepSeek always preferred the correct answer. If only whether the final answer was correct is counted, the differences among these inputs would be difficult to reflect. Seed also found a similar evaluation blind spot in the paper. In a control experiment with a self-trained 0.6B model, the average retrieval accuracy of the two models using full attention and block cache compression was 61.1% and 59.4%, respectively, a difference of only 1.7 percentage points. But after splitting the results by the position of the information, the situation was completely different. The position group where the full-attention model performed worst still had an accuracy of 58.4%; the block compression model, however, had only 9.9%. Two models with similar average scores may have completely different reliability at certain positions. Figure 3 | Similar average scores, but vastly different worst-position performance (chart based on the 0.6B experimental data from the Seed paper). In 2024, NVIDIA launched RULER and found that a model being able to pass a simple "needle in a haystack" test does not mean it can complete complex information retrieval and summarization. In 2025, Adobe and other institutions proposed NoLiMa and found that after reducing keyword overlap between questions and answers, model performance may also drop significantly. From "how many Tokens can it hold" to "can it accurately retrieve and understand information", long-context evaluation has continuously become more granular. And Seed's research this time revealed a new dimension: the compression architecture adopted by the model may also make certain fixed positions retrieval weaknesses. This difference is especially noteworthy in Agent scenarios. A Coding Agent needs to continuously read code, logs, and tool feedback, and the positions of key information will keep changing as the task progresses. A single retrieval error may affect subsequent operations. However, Seed's current experiments cannot directly prove how many real Agent task failures are caused by phase sensitivity; this still requires tests closer to actual workflows. Therefore, for models using block compression, in addition to average scores, it is also possible to further compare performance at different phases, accuracy at the worst positions, and whether these differences affect the completion of continuous tasks. For other architectures, different test methods may need to be designed.

06

Conclusion: An architecture race that has not yet ended. Looking back at this research, DeepSeek applied a new compression architecture to an actual model, while ByteDance Seed used public weights to find performance weaknesses that were previously difficult to notice. And before the paper was published, DeepSeek had already, through a new round of iteration, significantly reduced the relevant positional differences. This is also a noteworthy aspect of the current competition in large model technology: one company's architectural exploration may become another company's research subject; before a technological breakthrough has been fully understood, new problems and directions for improvement have already emerged.