Databases
StackOverflow2013
Postsdbo
POParsing large XML files?
Body
 

Note that there are some explanatory texts on larger screens.

plurals
  1. POParsing large XML files?
    text
    copied!<p>I have 2 xml files 1 with 115mb size and another with 34mb size.</p> <p>Wiile reading file A there is 1 field called desc that relations it with file B where I retrieve the field id from file B where desc.file A is iqual to name.file B.</p> <p>file A is already too big then I have to search inside file B and it takes a very long time to complete.</p> <p>How could I speed up this proccess or what would be a better approch to do it ?</p> <p>current code I am using:</p> <pre><code>#!/usr/bin/perl use strict; use warnings; use XML::Simple qw(:strict XMLin); my $npcs = XMLin('Client/client_npcs.xml', KeyAttr =&gt; { }, ForceArray =&gt; [ 'npc_client' ]); my $strings = XMLin('Client/client_strings.xml', KeyAttr =&gt; { }, ForceArray =&gt; [ 'string' ]); my ($nameid,$rank); open (my $fh, '&gt;&gt;', 'Output/npc_templates.xml'); print $fh "&lt;?xml version=\"1.0\" encoding=\"UTF-8\"?&gt;\n&lt;npc_templates xmlns:xsi=\"http://www.w3.org/2001/XMLSchema-instance\" xsi:noNamespaceSchemaLocation=\"npcs.xsd\"&gt;\n"; foreach my $npc ( @{ $npcs-&gt;{npc_client} } ) { if (defined $npc-&gt;{desc}) { foreach my $string (@{$strings-&gt;{string}}) { if (defined $string-&gt;{name} &amp;&amp; $string-&gt;{name} =~ /$npc-&gt;{desc}/i) { $nameid = $string-&gt;{id}; last; } } } else { $nameid = ""; } if (defined $npc-&gt;{hpgauge_level} &amp;&amp; $npc-&gt;{hpgauge_level} &gt; 25 &amp;&amp; $npc-&gt;{hpgauge_level} &lt; 28) { $rank = 'LEGENDARY'; } elsif (defined $npc-&gt;{hpgauge_level} &amp;&amp; $npc-&gt;{hpgauge_level} &gt; 21 &amp;&amp; $npc-&gt;{hpgauge_level} &lt; 23) { $rank = 'HERO'; } elsif (defined $npc-&gt;{hpgauge_level} &amp;&amp; $npc-&gt;{hpgauge_level} &gt; 10 &amp;&amp; $npc-&gt;{hpgauge_level} &lt; 15) { $rank = 'ELITE'; } elsif (defined $npc-&gt;{hpgauge_level} &amp;&amp; $npc-&gt;{hpgauge_level} &gt; 0 &amp;&amp; $npc-&gt;{hpgauge_level} &lt; 11) { $rank = 'NORMAL'; } else { $rank = $gauge; } print $fh qq|\t&lt;npc_template npc_id="$npc-&gt;{id}" name="$npc-&gt;{name}" name_id="$nameid" height="$npc-&gt;{scale}" rank="$rank" tribe="$npc-&gt;{tribe}" race="$npc-&gt;{race_type}" hp_gauge="$npc-&gt;{hpgauge_level}"/&gt;\n|; } print $fh "&lt;/&lt;npc_templates&gt;"; close($fh); </code></pre> <p>example of file A.xml:</p> <pre><code>&lt;?xml version="1.0" encoding="utf-16"?&gt; &lt;npc_clients&gt; &lt;npc_client&gt; &lt;id&gt;200000&lt;/id&gt; &lt;name&gt;SkillZone&lt;/name&gt; &lt;desc&gt;STR_NPC_NO_NAME&lt;/desc&gt; &lt;dir&gt;Monster/Worm&lt;/dir&gt; &lt;mesh&gt;Worm&lt;/mesh&gt; &lt;material&gt;mat_mob_reptile&lt;/material&gt; &lt;show_dmg_decal&gt;0&lt;/show_dmg_decal&gt; &lt;ui_type&gt;general&lt;/ui_type&gt; &lt;cursor_type&gt;none&lt;/cursor_type&gt; &lt;hide_path&gt;0&lt;/hide_path&gt; &lt;erect&gt;1&lt;/erect&gt; &lt;bound_radius&gt; &lt;front&gt;1.200000&lt;/front&gt; &lt;side&gt;3.456000&lt;/side&gt; &lt;upper&gt;3.000000&lt;/upper&gt; &lt;/bound_radius&gt; &lt;scale&gt;10&lt;/scale&gt; &lt;weapon_scale&gt;100&lt;/weapon_scale&gt; &lt;altitude&gt;0.000000&lt;/altitude&gt; &lt;stare_angle&gt;75.000000&lt;/stare_angle&gt; &lt;stare_distance&gt;20.000000&lt;/stare_distance&gt; &lt;move_speed_normal_walk&gt;0.000000&lt;/move_speed_normal_walk&gt; &lt;art_org_move_speed_normal_walk&gt;0.000000&lt;/art_org_move_speed_normal_walk&gt; &lt;move_speed_normal_run&gt;0.000000&lt;/move_speed_normal_run&gt; &lt;move_speed_combat_run&gt;0.000000&lt;/move_speed_combat_run&gt; &lt;art_org_speed_combat_run&gt;0.000000&lt;/art_org_speed_combat_run&gt; &lt;in_time&gt;0.100000&lt;/in_time&gt; &lt;out_time&gt;0.500000&lt;/out_time&gt; &lt;neck_angle&gt;90.000000&lt;/neck_angle&gt; &lt;spine_angle&gt;10.000000&lt;/spine_angle&gt; &lt;ammo_bone&gt;Bip01 Head&lt;/ammo_bone&gt; &lt;ammo_fx&gt;skill_stoneshard.stoneshard.ammo&lt;/ammo_fx&gt; &lt;ammo_speed&gt;50&lt;/ammo_speed&gt; &lt;pushed_range&gt;0.000000&lt;/pushed_range&gt; &lt;hpgauge_level&gt;3&lt;/hpgauge_level&gt; &lt;magical_skill_boost&gt;0&lt;/magical_skill_boost&gt; &lt;attack_delay&gt;2000&lt;/attack_delay&gt; &lt;ai_name&gt;SummonSkillArea&lt;/ai_name&gt; &lt;tribe&gt;General&lt;/tribe&gt; &lt;pet_ai_name&gt;Pet&lt;/pet_ai_name&gt; &lt;sensory_range&gt;15.000000&lt;/sensory_range&gt; &lt;/npc_client&gt; &lt;/npc_clients&gt; </code></pre> <p>example of file B.xml:</p> <pre><code>&lt;?xml version="1.0" encoding="utf-16"?&gt; &lt;strings&gt; &lt;string&gt; &lt;id&gt;350000&lt;/id&gt; &lt;name&gt;STR_NPC_NO_NAME&lt;/name&gt; &lt;body&gt; &lt;/body&gt; &lt;/string&gt; &lt;/strings&gt; </code></pre>
 

Querying!

 
Guidance
An individual column

Larger individual text columns get their own page to allow for proper reading.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload