IPCC Namespace — MediaWiki Word Count

Counts the total number of words published across all pages in the IPCC namespace (3000) of the ClimateKG production MediaWiki instance, using the MediaWiki Action API.

Instructions: 1. Set API_URL below to match your target environment. 2. Run all cells — the API calls take ~30–60 s for 99 pages. 3. Quarto renders from the stored cell outputs; no re-execution needed on quarto render.

Data source: https://prod-climatekg.semanticclimate.org/w/api.php
Namespace: IPCC (3000)
Method: action=query&list=allpages then prop=revisions&rvslots=main per page batch, with full continue-token handling to avoid response-size truncation.

Import Required Libraries

Configuration

Set API_URL to match your target environment.

Environment	API URL
LOCAL	http://localhost:8080/w/api.php
DEV	https://dev-climatekg.semanticclimate.org/w/api.php
TEST	https://test-climatekg.semanticclimate.org/w/api.php
PROD	https://prod-climatekg.semanticclimate.org/w/api.php

Connect to MediaWiki API

Helper functions that handle pagination and response-size continuation tokens.

Fetch All Pages in the IPCC Namespace

Retrieves all page titles from namespace 3000, then fetches wikitext content in batches of 10, handling API pagination at both stages.

# Step 1 — retrieve all page titles in the IPCC namespace
pages = get_all_page_titles(IPCC_NS)
titles_all = [p["title"] for p in pages]
print(f"Pages found in namespace {IPCC_NS}: {len(pages)}")

# Step 2 — fetch wikitext content in batches
content_map: dict[str, str] = {}
for i in range(0, len(titles_all), BATCH_SIZE):
    batch = titles_all[i : i + BATCH_SIZE]
    content_map.update(fetch_content_batch(batch))
    print(f"  fetched {min(i + BATCH_SIZE, len(titles_all))}/{len(titles_all)} pages", end="\r")
    time.sleep(REQUEST_DELAY)

print(f"\nPages with wikitext content : {len(content_map)}")
print(f"Pages empty / redirect      : {len(pages) - len(content_map)}")

Pages found in namespace 3000: 99
  fetched 99/99 pages
Pages with wikitext content : 99
Pages empty / redirect      : 0

Compute Word Count Per Page

Strips MediaWiki markup (templates, tables, links, HTML tags, list markers) then counts whitespace-separated tokens.

Summary Statistics

df_content = df[df["Has Content"]]

total_words   = df_content["Word Count"].sum()
total_pages   = len(df)
pages_content = len(df_content)
pages_empty   = total_pages - pages_content
mean_wc       = int(df_content["Word Count"].mean())
median_wc     = int(df_content["Word Count"].median())
max_wc        = df_content["Word Count"].max()
min_wc        = df_content["Word Count"].min()
largest_page  = df_content.loc[df_content["Word Count"].idxmax(), "Short Title"]
smallest_page = df_content.loc[df_content["Word Count"].idxmin(), "Short Title"]

summary_html = f"""
<table style="border-collapse:collapse;font-family:inherit;font-size:.95rem;margin-bottom:1rem">
  <tr style="background:#f0f0f0"><th colspan="2" style="padding:.5rem 1rem;text-align:left">
    IPCC Namespace (3000) — Word Count Summary
  </th></tr>
  <tr><td style="padding:.35rem 1rem">Total pages in namespace</td>
      <td style="padding:.35rem 1rem;font-weight:bold">{total_pages:,}</td></tr>
  <tr style="background:#f9f9f9"><td style="padding:.35rem 1rem">Pages with content</td>
      <td style="padding:.35rem 1rem;font-weight:bold">{pages_content:,}</td></tr>
  <tr><td style="padding:.35rem 1rem">Pages empty / placeholder</td>
      <td style="padding:.35rem 1rem;font-weight:bold">{pages_empty:,}</td></tr>
  <tr style="background:#e8f4e8"><td style="padding:.35rem 1rem"><strong>Total word count</strong></td>
      <td style="padding:.35rem 1rem;font-weight:bold;font-size:1.1rem">{total_words:,}</td></tr>
  <tr><td style="padding:.35rem 1rem">Mean words per page</td>
      <td style="padding:.35rem 1rem">{mean_wc:,}</td></tr>
  <tr style="background:#f9f9f9"><td style="padding:.35rem 1rem">Median words per page</td>
      <td style="padding:.35rem 1rem">{median_wc:,}</td></tr>
  <tr><td style="padding:.35rem 1rem">Largest page</td>
      <td style="padding:.35rem 1rem">{largest_page} ({max_wc:,} words)</td></tr>
  <tr style="background:#f9f9f9"><td style="padding:.35rem 1rem">Smallest page (with content)</td>
      <td style="padding:.35rem 1rem">{smallest_page} ({min_wc:,} words)</td></tr>
</table>
"""
display(HTML(summary_html))

IPCC Namespace (3000) — Word Count Summary
Total pages in namespace	99
Pages with content	99
Pages empty / placeholder	0
Total word count	7,524,958
Mean words per page	76,009
Median words per page	82,214
Largest page	SR15/Chapter-3 (168,790 words)
Smallest page (with content)	IPCC:AR6 (58 words)

# Words by report group — bar chart
group_totals = (
    df_content.groupby("Report Group")["Word Count"]
    .sum()
    .sort_values(ascending=False)
    .reset_index()
)

fig = px.bar(
    group_totals,
    x="Report Group",
    y="Word Count",
    title="Total Words by IPCC Report Group",
    labels={"Word Count": "Total Words", "Report Group": "Report Group"},
    color="Report Group",
    text="Word Count",
)
fig.update_traces(texttemplate="%{text:,.0f}", textposition="outside")
fig.update_layout(
    showlegend=False,
    yaxis_tickformat=",",
    uniformtext_minsize=9,
    uniformtext_mode="hide",
    margin=dict(t=60, b=40),
)
# Render as text/html so Quarto can display it from stored outputs
display(HTML(fig.to_html(full_html=False, include_plotlyjs="cdn")))

Word Count Table — All IPCC Namespace Pages

All 99 pages in namespace 3000, sorted by word count (descending). Pages with zero words are empty placeholders not yet populated with content.

table_df = df[["Short Title", "Report Group", "Word Count", "Has Content"]].copy()
table_df.columns = ["Page", "Report Group", "Word Count", "Has Content"]
table_df["Has Content"] = table_df["Has Content"].map({True: "Yes", False: "No"})

fig_table = go.Figure(
    data=[go.Table(
        columnwidth=[300, 100, 120, 100],
        header=dict(
            values=["<b>Page</b>", "<b>Report Group</b>",
                    "<b>Word Count</b>", "<b>Has Content</b>"],
            fill_color="#4472C4",
            font=dict(color="white", size=13),
            align="left",
            height=32,
        ),
        cells=dict(
            values=[
                table_df["Page"].tolist(),
                table_df["Report Group"].tolist(),
                [f"{v:,}" for v in table_df["Word Count"]],
                table_df["Has Content"].tolist(),
            ],
            fill_color=[
                [
                    "#e8f4e8" if has == "Yes" else "#fff3cd"
                    for has in table_df["Has Content"]
                ]
            ],
            align="left",
            font=dict(size=12),
            height=26,
        ),
    )]
)
fig_table.update_layout(
    title="IPCC Namespace Pages — Word Count (sorted by word count, descending)",
    margin=dict(t=60, b=10, l=10, r=10),
    height=100 + 28 * len(table_df),
)
# Render as text/html so Quarto can display it from stored outputs
display(HTML(fig_table.to_html(full_html=False, include_plotlyjs="cdn")))