2011-07-31

The real story behind the Java 7 GA bugs affecting Apache Lucene / Solr

I started this blog about two months ago, but until now, I had no time to write any posts. Since Oracle's release of Java 7 on Thursday last week, a lot of myths around the problem with Porter stemmer, wrong calculations in pulsing codec, and corrupt indexes appeared on the web. Some guys already blamed the Apache Software Foundation because they may have used the hype around Java 7 to fight against Oracle (since they stepped out of JCP).

I want to start with a chronological summary about what happened since last weekend:

Last Saturday I woke up in the morning and had no real plans for the weekend. I checked my Lucene JIRA issues and after I helped my GSoC student, I read Heise newsticker. One news article stepped into my eye: "Gearing up for Java 7" (english version). I noticed that there is only one week left until Oracle plans to release Java 7. "Maybe we should add some Jenkins jobs to the Apache build server to test Apache Lucene/Solr trunk and 3.x branches?" As always, Robert and me had some quick Google Talk session and I pointed out my plans. Our first problem was that the Jenkins server of Lucene runs under FreeBSD, so the chances were high, that we get no recent release of OpenJDK 7 running. I already installed the 64bit Windows preview build from Oracle (b147) on my Thinkpad and checked out the Lucene Core tests, which ran fine.

A quick review showed that OpenJDK7 build 147 was already available in the FreeBSD ports collection. I logged onto the jenkins slave machine and started to update ports and built the openjdk7 package. In parallel I also rebuilt the openjdk6 package for the standard test runs (a fix for a IPv6 bug was already available, so we were able to use latest version). After approx 25 minutes, the build was done and the package installed. I cloned the Jenkins jobs for the half-hourly test builds (3.x and trunk), hacked the shell scripts a little bit and started the first build. Robert and me were watching the build output,... live.

During the tests of the new analyzers module suddenly we got a crash with SIGSEGV in TestPorterStemmer. As I was only running the core tests on my local machine, this was not yet discoverered. We were shocked, "this is just a stupid FreeBSD problem, I am sure... hmhmhm... but a few weeks ago a user on the Solr mailing list reported a crash in PorterStemmer, too" (see this post). I started the analyzer tests locally on my windows box, bumm - same issue. Important was that the issue only appeared when I raised the number of test iterations, with the default of 1, the test passed. That clearly shows, there was an hotspot optimization bug, as they are only triggered when a method is executed ten thousand times.

I stopped the 2-hourly builds with Java 7 on the Jenkins slave, as we did not want to spam the mailing list with failed build reports. Robert opened LUCENE-3335 and started to investigate the problem. Step for step, he disabled compilation of several porter stemmer methods until he reached PorterStemmer.ends(). Robert opened a bug report at Oracles bug tracker: #7070134

The rest of the day we also fixed another small issues in our test cases (TestWordDelimiterFilter failed because it used a character that changed properties in Java 7 - Robert committed a fix for this).

On Sunday morning I gave Jenkins another chance: To proceed with testing on Jenkins, I hacked the shell scripts again to pass the following args to the Lucene test suite (only for Java 7 builds): -Dargs="-XX:CompileCommand=exclude,org/apache/lucene/analysis/en/PorterStemmer,ends"

The Lucene builds then passed fine most of the time, but we had some additional failures in another test inside the new facetting module: LUCENE-3346 (see below). But when starting to test the Solr builds, the problems began again: Some tests sometimes failed with strange error messages, but only randomly. It took us two hours and a hint from Yonik to find the issue behind this (SOLR-2673):
"It looks like with Java7, that the test methods are not being run in the order they are declared in the file. That's probably the root cause of the other Solr test failures with Java7, too."
A quick test with Java 6, inserting a Collections.shuffle() into the customized Lucene TestRunner, produced the same bugs. Additionally, the JDK docs nowhere explicitely state that Class.getMethods() does return in a particular order:
"Returns an array containing Method objects reflecting all the public member methods of the class or interface represented by this Class object, including those declared by the class or interface and those inherited from superclasses and superinterfaces. Array classes return all the (public) member methods inherited from the Object class. The elements in the array returned are not sorted and are not in any particular order."
The next few hours were then spent in fixing all Solr tests that relied on order of method executions (which is a violation against testing principles). On the evening we had all tests running fine, except the facetting bug and a random failure in the ICU module: LUCENE-3344 (I will come back to that later).

Status on Sunday: Until now, no response from Oracle about the new bug...

Monday morning CET, still no news from Oracle!

In late afternoon we got the first response; Robert informed me:
"Bug is visible, but it's priority is LOW. I am sure, they will release Java 7 with PorterStemmer crashing. All Solr users with the default configuration will blame us!"
Dawid Weiss then contacted the hotspot-compiler-dev mailing list; I subscribed to it and started to ask them questions. The developer responsible for these bugs (Vladimir Kozlov) sent me in the evening links to patches that might fix the issue. I also got an explanation, why we had this issue, that also exists in Java 6, but is hidden there due to optimizations not enabled (please read the thread for more information).

On Tuesday morning CET I started to produce a FreeBSD ports patch, so I was able to compile our openjdk7 package on the Jenkins slave with the proposed fixes (see patch-0uwe.patch). I changed the Jenkins configuration again and suddenly all tests passed, even the facetting tests!

We then worked around the last Java7-related test bug (LUCENE-3344), which made ICU fail on classloading ULocale. The problem is caused by some new locales in Java 7, that lead to a chicken-and-egg problem in the static initializer of ULocale. It initializes its default locale from the JDK locale in a static ctor. Until the default ULocale instance is created, the default is not set in ULocale. But ULocale's ctor itsself needs the default locale to fetch some ressource bundles and throws NPE. We opened a bug report at ICU (#8734) and added a workaround into our LuceneTestCase's locale randomization.

Time goes on...

On Thursday, lunch time CET, Java 7 was released by Oracle. Still hoping for a wonder, I downloaded the Windows x64 build from the official download location. Clicking on install it reported: "this version of the JDK is already installed on your computer, do you want to reinstall?" I did this and I was able to verify: it's exactly the same version as released on June 27th (you can verify this, too: the timestamp on the signature of the Windows Installer EXE file is June 27th). So Oracle ignored all bugs (not only ours) in the preview release and simply released a one month old package! So what was the sense of the preview release? They could have released it one month before! It was for sure not intended for public review and bug hunting!

In the evening (CET) I then opened a new issue (LUCENE-3349) as we already discussed on IRC, that we don't want our users to crash their Solr installations and corrupt their indexes. Robert spent the last days on hunting the causes behind the CheckIndex failures in the facetting module (the so-called index corru(m)ption bug). He was able to produce some random seeds that triggered the bug. In fact, it is a reincarnation of the well known readVInt-bug (LUCENE-2975) we discovered before release of Lucene 3.1. It affects the most often used Lucene method during reading index contents from disk: DataInput.readVInt(). For optimization purposes we have several incarnations of this method dependent on the underlying data structures. Java 6 JDKs only triggered this bug in combination with MappedByteBuffer.getByte(), but now it was triggered almost everywhere! Especially when the new pulsing codec was used (which has a byte[] variant of this method), all was f*cked up! StandardPostingsReader was merging index segments, the wrong results of readVInt() were copied to the newly created index segments, and finally we produced corrupt indexes!

The warning mail was released...

Shortly before midnight CET I sent the warning mail, prepared in LUCENE-3349, to the mailing lists:
Hello Apache Lucene & Apache Solr users, Hello users of other Java-based Apache projects,

Oracle released Java 7 today. Unfortunately it contains hotspot compiler optimizations, which miscompile some loops. This can affect code of several Apache projects. Sometimes JVMs only crash, but in several cases, results calculated can be incorrect, leading to bugs in applications (see Hotspot bugs 7070134 [1], 7044738 [2], 7068051 [3]).

Apache Lucene Core and Apache Solr are two Apache projects, which are affected by these bugs, namely all versions released until today. Solr users with the default configuration will have Java crashing with SIGSEGV as soon as they start to index documents, as one affected part is the well-known Porter stemmer (see LUCENE-3335 [4]). Other loops in Lucene may be miscompiled, too, leading to index corruption (especially on Lucene trunk with pulsing codec; other loops may be affected, too - LUCENE-3346 [5]).

These problems were detected only 5 days before the official Java 7 release, so Oracle had no time to fix those bugs, affecting also many more applications. In response to our questions, they proposed to include the fixes into service release u2 (eventually into service release u1, see [6]). This means you cannot use Apache Lucene/Solr with Java 7 releases before Update 2! If you do, please don't open bug reports, it is not the committers' fault! At least disable loop optimizations using the -XX:-UseLoopPredicate JVM option to not risk index corruptions.

Please note: Also Java 6 users are affected, if they use one of those JVM options, which are not enabled by default: -XX:+OptimizeStringConcat or -XX:+AggressiveOpts

It is strongly recommended not to use any hotspot optimization switches in any Java version without extensive testing!

In case you upgrade to Java 7, remember that you may have to reindex, as the unicode version shipped with Java 7 changed and tokenization behaves differently (e.g. lowercasing). For more information, read JRE_VERSION_MIGRATION.txt in your distribution package!

On behalf of the Lucene project,
Uwe


What then happened is well-known:


Until now, California was still sleeping...


In late afternoon CET, California woke up:


On the evening before, Hoss already posted the following blog post on the Lucid Imagination website: "Don’t Use Java 7, For Anything".

  • After California woke up, the first person posted Hoss' blog post on Slashdot: "Java 7 Ships With Severe Bug" referring to Hoss' blog post.

And then the whole thing went crazy: On Twitter, new posts refering to Slashdot, InfoWorld, and, finally, Hoss' blog post on Lucid Imagination appeared every few seconds. There were more tweets stating "Don't use Java 7, For Anything" than tweets about "the first Java release since 4 years".

A little bit later I recognized the first pro-Oracle blog trying to explain some background information: "Don't Use Java 7? Are you kidding me?" (Markus Eisele). Thanks for posting this!

I went to sleep and on the following day, the original Oracle Bug report that caused this was upgraded to priority "HIGH" - yeah. So we will hopefully get a corrected Java 7 release quite soon in Update Pack 1!

Finally I wanted to say thank you to the other Lucene committers that helped during investigation: Robert Muir, Yonik Seeley, Dawid Weiss, and Mike McCandless. And of course Hoss for his funny caption on Lucid Imagination's blog!

Update: (2011-08-01)

On Saturday morning CET, Cay Horstmann, professor of computer science at San Jose State University, compared the JDK 7 bugs with the Pentium FDIV bug in 1994. In his blog article "Java 7 Unsafe at Any Speed?", he stated that SIGSEGVs are easy bugs; much more serious are hidden bugs, only appearing under certain conditions and then silently produce wrong computation results. On Sunday evening CET, a user asked on Stackoverflow: "How serious is the Java7 'Solr/Lucene' bug?" In his response, Robert Muir described his work how to track down Java's "pentium bug" and circumvent it, with no success (see above).

Luckily, Oracle raised the priority of the SIGSEGV bug (#7070134) to "HIGH", but the other two bugs are still on "MEDIUM". One of them (#7044738) is exactly such a bug that Cay Horstmann described in his blog post. We applied the patches for all three bugs to our JVM installation, the "HIGH" priority one only fixes the SIGSEGV. Since our wrong readVInt-calculations in the method's loop are fixed by the combination of all three patches, wouldn't it be a good idea for Oracle, to rate all three issues with priority "HIGH"? Otherwise it could happen that we get service pack 1 only with the SIGSEGV bug fixed, but still silently producing corrupt indexes?

On Monday, the German "JAXenter" / "IT Republic" published a nice article: "Wie gravierend sind die Bugs in JDK7 wirklich?", referring to the above posts. There is also an English version available: "Java 7 Causes Headaches for Lucene and Solr Users".

Update: (2011-08-03)

Yesterday I had some interviews with journalists/bloggers:

Update: (2011-08-04)

Today, Neil McAllister published "Oracle: Java's worst enemy" on the InfoWorld newsticker. When California woke up in the evening CET, someone posted this article on Slashdot with the title "Oracle's Java Policies Are Destroying the Community", resulting in a heated discussion and a high tweet rate again.

Update: (2011-08-10)

The last days, community and journalists were blogging about the backgrounds:
Two days ago, Oracle offered to some committers that they could get access to early builds of the Java SE before they are released. This would allow us to check compatibility of bugfix releases and service packs of the Java SE with Apache Lucene and Solr. All this is covered by their Java CAP (Compatibility and Performance Program).


Update: (2011-08-12)

Oracle published a FAQ about Java 7, that clarifies when and how the Lucene-related fixes in Java 7 will be released. They also mentioned another bug (#7077439), they investigated while fixing the others. Unfortunately the page is not accessible to the public, but the fix and an explanation is already reviewed and committed to OpenJDK. Does this mean, Oracle started to try and test Lucene builds with their JDK?

    134 comments:

    1. Thanks, nice summary to understand the Java7 bug history better. I'm curious how the Oracle story sounds. :-)

      ReplyDelete
      Replies
      1. Interesting thoughts. I advice you to post this thoughts in Instagram so that many interested people can see it. And you can always buy instagram followers to quickly increase their number.

        Delete
    2. "Lucene/SolrをJava 7で使うときの注意(あるいはJava 6以前でもホットスポットのバグを踏む可能性あり)"

      My Japanese is crappy, but here it goes: "Warning when you use Lucene/Solr with Java 7 (probably there is a possibility for Hotspot Bugs appearing in versions up to Java 6 as well)"

      BTW, interesting post!

      ReplyDelete
    3. Regarding the ordering issue in Class.getMethods (and getDeclaredMethods), you may want to look at and vote on http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6807471

      ReplyDelete
    4. Andrew: We recognized this behaviour for the reflection layer in Java 7, but it is no bug at all, the javadocs explicitely state that the returned arrays have no explicit order. The bug was in fact on our side, as the JUnit tests of Solr expected the array ordered like the method declarations in the source file (which was always the case in previous Java versions, but there was no guarantee).

      ReplyDelete
    5. Unlike you, I don´t think there´s anything to compliment in Hoss´ "creative" headline. Yes, it might have succeeded in getting a fix done sooner, but at what cost?.

      "And then the whole thing went crazy" is quite an understatement. I´m sure the foes of Java, you know the ones that wanted to kill Java since the 1.0 days, will use the worst-titled stories quoted above to discredit Java as a platform and promote their alternative solutions.

      "Don´t use Java 7, at all" is plain FUD... it prevents end-users on desktop computers from trying the new release which based on my tests gives much better performance.

      I´ve been using JRE7 to run the kind of desktop Java apps that I use daily (Java Image Editor, Vuze, muCommander withoout a single crash -so far, knock on wood ;). And I bet there´s quite a bit of loops in that code.

      Just my $0.02

      ReplyDelete
    6. Oh "Don´t use Java 7, at all" and the following backlash only conveys the message that "Oracle releases buggy code" and implicitely that "they don´t know what they´re doing" and are not to be trusted.

      It´s ironic that we don´t see headlines like "Microsoft releases buggy software" given Microsoft´s track record and the sole existance of "Patch tuesday".

      FC

      ReplyDelete
    7. Hossman did an awesome job. He probably prevented more cases of index corruption with one blog post than I ever have hammering on Lucene's tests.

      ReplyDelete
    8. That's fascinating! When will the movie be coming out? I wonder who they will get to play you...

      ReplyDelete
    9. Thank you for a great post. The movie - what stops you from doing a YouTube trailer? In this style: Hamlet

      ReplyDelete
    10. Great story! I can't see any point in blaming Oracle developers as well as anybody else.
      Had they had the bug fixed before the release, it would be better, but who knows their internal matters? At least they've done the release :)

      ReplyDelete
    11. So, a huge amount of time has passed. Is it now fixed?

      ReplyDelete
    12. You can read in the later blog posts that the bugs are fixed since Oracle Java 7u1 and 6u29.

      ReplyDelete
    13. Great post, and great website. Thanks for the information! ChoreRelief

      ReplyDelete
    14. What kinds of searches could direct another consumer for this content?. see Not every site allows you to build links that point to other sites.

      ReplyDelete
    15. I can’t imagine focusing long enough to research; much less write this kind of article. You’ve outdone yourself with this material. This is great content. House Cleaning

      ReplyDelete
    16. Learning how to do SEO can be a very time consuming task, and if you want to get up to a standard where you can claim to be an SEO "expert" you will have to put a lot of hours in. To be able to learn the discipline well, you will also need to have previous experience of web design, web content and marketing strategies, if you want to be able to understand some of the subtle nuances about what is required from you. You will need to be able to dedicate a lot of time to keeping up with the latest strategies, or else you could get left behind. Therefore becoming your own SEO expert requires more than just a one off time-cost. However, if you have enough time to do this, without detracting from your other business commitments, then you may be able to save yourself a little money on the costs of hiring professional SEO workers. seo company

      ReplyDelete
    17. Get this apk for awesome streaming service Hotstar Mod Apk

      Live matches can also be seen here

      ReplyDelete
    18. 훌륭한 포스트! 자세한 내용은 먹튀안전업체

      ReplyDelete
    19. Very interesting blog. A lot of blogs I see these days don't really provide anything that I'm interested in, but I'm most definitely interested in this one. 먹튀

      ReplyDelete
    20. I think this is an edifying post and it is useful and capable. I am always searching online for articles that can help me. I truly believe these site requirements additional consideration. I favor which a great deal. braces in tijuana

      ReplyDelete
    21. I was surfing net and fortunately came across this site and found very interesting stuff here. Its really fun to read. I enjoyed a lot. Thanks for sharing this wonderful information. list of developers in dubai

      ReplyDelete
    22. I am very impressed with your writing먹튀검증 I couldn't think of this, but it's amazing! I wrote several posts similar to this one, but please come and see!

      ReplyDelete
    23. Thank you! This is what I need to find. 토토사이트 Hope to hear more news from you.

      ReplyDelete
    24. Hi, thank you so much for sharing this, it is extremely useful. Thanks again 메이저놀이터

      ReplyDelete
    25. This website and I conceive this internet site is really informative ! Keep on putting up! 토토사이트

      ReplyDelete
    26. I truly like you're composing style, incredible data, thankyou for posting. 먹튀검증

      ReplyDelete
    27. Your article got me thinking about something. Who do we contact if we want anesthesia quickly? I have heard there are some mobile services online, and now I know which of these services is best at their job. I am glad my friends told me about them. 먹튀폴리스꽁머니

      ReplyDelete
    28. This is a great post. I like this topic.This site has lots of advantage.I found many interesting things from this site. It helps me in many ways.Thanks for posting this again. 안전놀이터

      ReplyDelete
    29. Very informative post ! There is a lot of information here that can help any business get started with a successful social networking campaign 먹튀검증

      ReplyDelete
    30. I like this post, enjoyed this one appreciate it for posting. “Abortion is advocated only by persons who have themselves been born.” by Ronald Reagan. 먹튀검증

      ReplyDelete
    31. Thanks for your nice post I really like it and appreciate it. My work is about Packaging Printers
      . If you need perfect quality boxes then you can visit our website.

      ReplyDelete
    32. What an interesting story! I'm glad I finally found what I was looking for 카지노사이트.

      ReplyDelete
    33. A very awesome blog post. 토토사이트 We are really grateful for your blog post. combat, law enforcement You will find a lot of approaches after visiting your post. I was exactly searching for. Thanks for such post and please keep it up. Great work.

      ReplyDelete
    34. A very awesome blog post. 강남출장

      We are really grateful for your blog post. combat, law enforcement You will find a lot of approaches after visiting your post. I was exactly searching for. Thanks for such post and please keep it up. Great work

      ReplyDelete
    35. Hello
      Thank you for giving me useful information.
      Please keep posting good information in the future
      I will visit you often. 토토사이트Thank you.
      I am also running the site. 메이저놀이터 This is a related site, so please visit once.안전놀이터
      Have a niceday!

      ReplyDelete
    36. When i expanded your website, I liked it a lot because it has a lot of useful information.My company provide quality packaging boxes in the United Kingdom.My boxes are available in all shapes and sizes.If anyone is interested in buying boxes,visit my Packaging Printers
      .
      Thank You!

      ReplyDelete
    37. First of all, thank you for your post. Your posts are neatly organized with the information I want, so there are plenty of resources to reference. I bookmark this site and will find your posts frequently in the future. Thanks again ^^ 바둑이사이트

      ReplyDelete
    38. Hello
      Thank you for giving me useful information.
      Please keep posting good information in the future
      I will visit you often. 토토사이트Thank you.
      I am also running the site. 메이저놀이터 This is a related site, so please visit once.안전놀이터
      Have a niceday!

      ReplyDelete
    39. When I read an article on this topic, 먹튀검증 the first thought was profound and difficult, and I wondered if others could understand.. My site has a discussion board for articles and photos similar to this topic. Could you please visit me when you have time to discuss this topic?

      ReplyDelete
    40. Easily, the article is actually the best topic on this registry related issue. I fit in with your 먹튀검증 and will eagerly look forward to your next updates. Just saying thanks will not just be sufficient, for the fantastic lucidity in your writing. I will instantly grab your rss feed to stay informed of any updates

      ReplyDelete
    41. I am overwhelmed by your post with such a nice topic. Usually I visit your 안전놀이터 and get updated through the information you include but today’s blog would be the most appreciable. Well done!

      ReplyDelete
    42. Your site is very good today. My work is according to you.If you want to check out my site please visit my site customized packaging custom boxes

      ReplyDelete
    43. Of course, your article is good enough, but I thought it would be much better to see professional photos and videos together. There are articles and photos on these topics on my homepage, so please visit and share your opinions. 토토커뮤니티

      ReplyDelete
    44. I am contemplating this topic. I think you can solve my problems. My site is at "카지노사이트". I hope you can help me.

      ReplyDelete
    45. I have gained a lot from reading this article. Please write a lot of articles related to 먹튀검증 in the future.

      ReplyDelete
    46. In the meantime, I wondered why I couldn't think of the answer to this simple problem like this. Your article is an article that gives the answer to all the content I've been contemplating. 온라인홀덤

      ReplyDelete
    47. Hello
      Thank you for giving me useful information.
      Please keep posting good information in the future
      I will visit you often. 토토사이트Thank you.
      I am also running the site. 메이저놀이터 This is a related site, so please visit once.안전놀이터
      Have a niceday! 레플리카

      ReplyDelete
    48. Looking at this article, I miss the time when I didn't wear a mask. 먹튀검증커뮤니티 Hopefully this corona will end soon. My blog is a blog that mainly posts pictures of daily life before Corona and landscapes at that time. If you want to remember that time again, please visit us.

      ReplyDelete
    49. The article composing business is going extraordinary for me at this moment, and the vast majority of my prosperity is because of the substance that you post here. I utilize your substance to compose my articles, and the substance makes my great article incredible. 바카라검증사이트

      ReplyDelete
    50. In the meantime, I wondered why I couldn't think of the answer to this simple problem like this. Your article is an article that gives the answer to all the content I've been contemplating. 메이저토토사이트

      ReplyDelete
    51. Good pictures and great post very informative thanks for sharing with us.
      Branding Mailer boxes
      Hanger boxes USA

      ReplyDelete
    52. I always think about what is. It seems to be a perfect article that seems to blow away such worries. 먹튀검증사이트 seems to be the best way to show something. When you have time, please write an article about what means!!

      ReplyDelete
    53. Wow, great blog article.Really looking forward to read more. Awesome.부산오피

      ReplyDelete
    54. One other issue is when you are in a predicament where you would not have a co-signer then you may genuinely wish to try to exhaust all of your financing options. You can get many grants or loans and other grants that will give you finances to assist with school expenses. Thank you for the post.
      egaon casino

      ReplyDelete
    55. It's really a nice and useful piece of info. I'm satisfied that you simply shared this useful info with
      us. Please keep us up to date like this. Thanks for sharing. 부산오피


      ReplyDelete
    56. Exceptional post however , I was wanting to know if you could write a litte more on this topic? I’d be very thankful if you could elaborate a little bit further. Thanks 사설토토사이트

      ReplyDelete
    57. Attractive section of content. I simply stumbled upon your blog and in accession capital to assert that I acquire in fact enjoyed
      account your blog posts. Anyway I’ll be subscribing
      on your augment or even I success you access constantly quickl

      Also visit my site:スロット


      ReplyDelete
    58. I’m truly enjoying the design and layout of your blog. It’s a very easy on the eyes which makes it much more pleasant for me to come here and visit more often. Did you hire out a designer to create your theme? Great work! ufabet1688

      ReplyDelete
    59. I'm so happy to finally find a post with what I want. 안전놀이터순위 You have inspired me a lot. If you are satisfied, please visit my website and leave your feedback.

      ReplyDelete
    60. this type of article will be able to think and analyzed to the readers thats leads them in a bright future.
      파워볼

      ReplyDelete
    61. 바카라사이트
      it makes me feel that our country has a more things nned to improve in terms of managing and good things happen .

      ReplyDelete
    62. this article would help us as a citizens in this country how make it more better.
      스포츠토토

      ReplyDelete
    63. cant imagine if those things would disappear and eventually gone.
      바카라사이트

      ReplyDelete
    64. Of course, your article is good enough, 먹튀검증 but I thought it would be much better to see professional photos and videos together. There are articles and photos on these topics on my homepage, so please visit and share your opinions.

      ReplyDelete
    65. Good morning!! I am also blogging with you. In my blog, articles related to are mainly written, and they are usually called 메이저사이트. If you are curious about , please visit!!

      ReplyDelete
    66. Hello, I'm happy to see some great articles on your site. Would you like to come to my site later? My site also has posts, comments and communities similar to yours. Please visit and take a look 토토사이트

      ReplyDelete
    67. Hello, I'm happy to see some great articles on your site. Would you like to come to my site later? My site also has posts, comments and communities similar to yours. Please visit and take a look 메이저놀이터

      ReplyDelete
    68. Just how can you compose a composing similar to this? I are just one of individuals that creates as well as uploads, however I liked your writing due to the fact that there were many points to mimic. I desire you to see my writing and also have a look at it. I can not compose like you, however thanks for analysis and also offering me responses. 바카라사이트

      ReplyDelete
    69. I'm writing on this topic these days, 메이저놀이터추천, but I have stopped writing because there is no reference material. Then I accidentally found your article. I can refer to a variety of materials, so I think the work I was preparing will work! Thank you for your efforts.

      ReplyDelete
    70. I got a web site from where I be capable of really obtain valuable information regarding my study and knowledge.
      Great Article… Good Job… Thanks For Sharing…

      Website:바카라

      ReplyDelete
    71. I’m not sure exactly why but this weblog is loading incredibly slow for me. Is anyone else having this problem or is it a problem on my end? I’ll check back later on and see if the problem still exists. 안전놀이터순위

      ReplyDelete

    72. Attractive section of content. I simply stumbled upon your blog and in accession capital to assert that I acquire in fact enjoyed
      account your blog posts. Anyway I’ll be subscribing
      on your augment or even I success you access constantly quickly.

      Here's my website : -- 안마
      (jk)

      ReplyDelete
    73. Deadlines are often scary. Aren’t they? If you the same then you are in need of someone who can wipe this fear away. And professional assignment help can do this thing in such a smooth way that you will not even realize. All you have to do is to carry out a little bit of research on the web.

      ReplyDelete
    74. It's really a nice and useful piece of info. I'm satisfied that you simply shared this useful info with
      us. Please keep us up to date like this. Thanks for sharing.안마


      ReplyDelete
    75. Thanks for your marvelous posting! I actually enjoyed reading it, you could be
      a great author.I will remember to bookmark your blog and will
      eventually come back from now on. I want to encourage you to continue your great
      writing, have a nice weekend!

      Website:바카라사이트


      ReplyDelete
    76. Jeewangarg is a renowned Website Designing Company in Delhi offering the ultimate designing and developing services. This website designing company has a team of expert’s designer and developers equipped to handle an array of in-house services. JeewanGarg is the website designing expert company in Delhi drafts a website that goes beyond aesthetics to include the website’s all-inclusive performance.

      ReplyDelete
    77. While asking yourself ‘Who will do my assignment online?’, students may have doubt about whether someone will find out that you delegated the assignment to someone else. Students should not worry at all. When you choose us, you will enjoy freedom from fear of someone revealing that you received help. We strive to keep our Students anonymous by keeping their information safe and not sharing it with third parties. Even our writers don’t know your identity.

      ReplyDelete
    78. You have done a great job on this article. It’s very readable and highly intelligent. You have even managed to make it understandable and easy to read. You have some real writing talent. Thank you. 부산달리기


      ReplyDelete
    79. Digital Marketing Company In Bangalore, India: DigiMark Agency is one of the Best Digital Marketing companies in Bangalore. Ranked among the Top digital marketing agencies in Bangalore. https://www.digimarkagency.com/

      ReplyDelete
    80. Personal Loan Provider In Jaipur: Apply for an online instant personal loan in Jaipur with instant approval, low-interest rate, and minimum documentation at NiraFinance. https://nirafinance.com/personal-loan-in-jaipur

      ReplyDelete
    81. ABBS SM is Bangalore's one of the top Bangalore Business School and best m ba pgdm colleges in bangalore. It is know for its distinctive teaching With 100% Placement Assistance Top PGDM Colleges in Bangalore |
      top business schools in bangalore

      ReplyDelete
    82. According to the writers of Masters dissertation writing service, this skill is most wanted and through this, professionals can make their future secure and spend a successful life.

      ReplyDelete
    83. GPS Clock : Essae's Digital GPS Clock is manufactured in RED LED Display in different size, different frequency range and with 32 time zones.
      GPS Digital Clock

      ReplyDelete
    84. Metro Mane - Building your Dreams
      Interior Construction Contractor | Building Contractors Bangalore
      House Building Contractors: Metro Mane is a one stop solution for all your building related services. Founded by group of experienced engineers to provide customer best quality and safe execution.
      building construction contractors in bangalore

      ReplyDelete
    85. สมัครยูฟ่าเบท Wow, incredible blog layout! How long have you been blogging for? you make blogging look easy. The overall look of your web site is great, let alone the content!

      ReplyDelete
    86. Abbs is one of the top bca colleges in bangalore . ABBS BCA program has an excellent group of dedicated faculty & experts from the IT industry, guiding the students

      ReplyDelete
    87. I've seen articles on the same subject many times, but your writing is the simplest and clearest of them. I will refer to 메이저놀이터추천

      ReplyDelete
    88. Wonderful beat ! I wish to apprentice while you amend
      your web site, how can i subscribe for a blog site? The account helped me a acceptable deal.
      I had been a little bit acquainted of this your broadcast
      provided bright clear idea.
      Here is my web site 부산오피

      ReplyDelete
    89. Thanks for your marvelous posting! I actually enjoyed reading it, you could be
      a great author.I will remember to bookmark your blog and will
      eventually come back from now on. I want to encourage you to continue your great
      writing, have a nice weekend!

      Website:카지노

      ReplyDelete
    90. I’m very pleased to discover this site. I want to to thank you for ones time for this particularly wonderful read!! I definitely savored every part of it and i also have you saved as a favorite to see new information on your blog. 메이저토토사이트

      ReplyDelete
    91. It’s appropriate time to make some plans for the future and it’s time to be happy. I have read this post and if I could I wish to suggest you some interesting things or tips. Maybe you could write next articles referring to this article. I wish to read more things about it! สมัครยูฟ่าเบท

      ReplyDelete
    92. Nice post. I was checking constantly this blog and I’m impressed! Extremely useful info specially the last part I care for such information a lot. I was seeking this certain info for a long time. Thank you and good luck. ยูฟ่า168

      ReplyDelete
    93. Thanks for your marvelous posting! I actually enjoyed reading it, you could be
      a great author.I will remember to bookmark your blog and will
      eventually come back from now on. I want to encourage you to continue your great
      writing, have a nice weekend!
      카지노사이트

      ReplyDelete

    94. I used to be suggested this blog by means of my cousin.
      I am not sure whether or not this put up is
      written by means of him as nobody else recognize such unique about my problem.
      You are wonderful! Thank you!

      My web page::; 강남안마

      (freaky)

      ReplyDelete
    95. This is a really good article. I recently discovered your blog and wanted to express my gratitude for taking the time to go through your postings. After all, I’ll be subscribing to your feed and hoping to hear from you soon! To know about Best Essay Writing Help, visit at https://www.allassignmenthelp.co.uk/

      ReplyDelete
    96. I'm so happy to finally find a post with what I want. 실시간바카라 You have inspired me a lot. If you are satisfied, please visit my website and leave your feedback.

      ReplyDelete
    97. Been looking for this article for long time ago and finally found here. Thanks for the tips and sharing this post. appreciate!Please check out my website too and let me know what you think.
      바카라사이트

      ReplyDelete
    98. Does your home need some sprucing up? Maybe some old drywall needs fixing, give us a call and let us give you a FREE estimate. Small project? No problem! Colorado Springs Drywall Contractor

      ReplyDelete
    99. Prakash Tradewell, with over decades of experience in manufacturing and supplying electrical insulating sleeves and cables, has boarded on new industry benchmarks. Our range includes the best fibre sleeves, heat-shrinkable sleeves, fiberglass sleeve, insulating wires, fiber glass manufacturer in india. We also offer polyester yarns and polyester threads. Prakash Tradewell’s procure high-quality raw material from our reliable vendors for the manufacturing of these products.

      ReplyDelete
    100. Youre so right. Im there with you. Your weblog is definitely worth a read if anyone comes throughout it. Im lucky I did because now Ive received a whole new view of this. 먹튀검증사이트


      ReplyDelete
    101. I am always on the lookout for quality posts and articles so i suppose im lucky to have found this! I hope you will be adding more in the future 토토

      ReplyDelete
    102. This is such a great resource that you are providing and you give it away for free. I love seeing blog that understand the value. Im glad to have found this post as its such an interesting one!
      경마사이트
      경마

      ReplyDelete
    103. Common outings let us discuss simplest way to thanks a ton for your personal efforts, which will is why I’m just checking out the positioning on a daily basis, interested in innovative, helpful information and facts. Lots of, thanks a lot! 카지노

      ReplyDelete
    104. Pretty good post. I just stumbled upon your blog and wanted to say that I have really enjoyed reading your blog posts. Any way I’ll be subscribing to your feed and I hope you post again soon. Big thanks for the useful info. 토토

      ReplyDelete
    105. Hello, I read the post well. 안전놀이터추천 It's a really interesting topic and it has helped me a lot. In fact, I also run a website with similar content to your posting. Please visit once


      ReplyDelete
    106. I think it will be more complete if you add some topics to this article. I have these types of articles on my blog.
      토토사이트

      ReplyDelete
    107. This is a great post, I am very happy with this information.
      사설토토

      ReplyDelete
    108. I’m impressed, I must say. Actually rarely must i encounter a blog that’s both educative and entertaining, and let me tell you, you could have hit the nail on the head. Your notion is outstanding; the issue is an issue that too little individuals are speaking intelligently about. ufabet

      ReplyDelete
    109. Nice to be visiting your blog again, it has been months for me. Well this article that i've been waited for so long. I need this article to complete my assignment in the college, and it has same topic with your article. Thanks, great share. ufabet1688

      ReplyDelete
    110. very great eccommerce website for custom packaging boxes try now branded boxes uk

      ReplyDelete
    111. This was really an interesting topic and I kinda agree with what you have mentioned here!Hello, I have browsed most of your posts. This post is probably where I got the most useful information for my research. Thanks for posting, maybe we can see more on this. Are you aware of any other websites on this subjec 에볼루션카지노

      ReplyDelete
    112. This is such a great resource that you are providing and you give it away for free. I love seeing blog that understand the value. Im glad to have found this post as its such an interesting one! I am always on the lookout for quality posts and articles so i suppose im lucky to have found this! I hope you will be adding more in the future. 토토사이트추천

      ReplyDelete
    113. This was really an interesting topic and I kinda agree with what you have mentioned here!Hello, I have browsed most of your posts. and good major info for you 카지노사이트

      ReplyDelete
    114. Nice to be visiting your blog again, it has been months for me. Well this article that i've been waited for so long. I need this article info for major site info for you 메이저사이트

      ReplyDelete
    115. looking for this article for long time ago and finally found here and good major site info for come my web site메이저사이트

      ReplyDelete
    116. Very well written information. It will be beneficial to anybody who utilizes it, including me. Keep doing what you are doing – for sure i will check out more posts. ufabet

      ReplyDelete
    117. Shot Blasting is a procedure to clean, strengthen, or polish metal. Shot Blast Machine is required for shot blasting in almost every industry that uses metal, including automotive, aerospace, foundry, construction, rail, shipbuilding, and many alike. Shot Blasting Technique is used for surface protection and prior-preparation surfaces to further processing such as welding, painting, etc.
      We offer world-class and efficient long-term solutions and complete service features for users of shot blasting technique. These also include new Shot Blasting Machines along with the customized solution. Our experienced team uses its impeccable expertise in the enhancement, design, manufacturing, and distribution of Shot Blasting Equipment.

      ReplyDelete
    118. I am really feel glad to read out published post on this blog. Apart from this, I am still waiting other post for escalating my knowledge power. In the time scarcity situation, you can leverage from Assignment Help Service to secure highest grade.

      ReplyDelete